Robotic fruit picking system

ABSTRACT

A robotic fruit picking system includes an autonomous robot that includes a positioning subsystem that enables autonomous positioning of the robot using a computer vision guidance system. The robot also includes at least one picking arm and at least one picking head, or other type of end effector, mounted on each picking arm to either cut a stem or branch for a specific fruit or bunch of fruits or pluck that fruit or bunch. A computer vision subsystem analyses images of the fruit to be picked or stored and a control subsystem is programmed with or learns picking strategies using machine learning techniques. A quality control (QC) subsystem monitors the quality of fruit and grades that fruit according to size and/or quality. The robot has a storage subsystem for storing fruit in containers for storage or transportation, or in punnets for retail.

CROSS-REFERENCE TO RELATED APPLICATIONS

This is a continuation of copending application Ser. No. 16/406,505,filed on May 8, 2019, which is a continuation of PCT Application No.PCT/GB2017/053367, filed on Nov. 8, 2017, which claims priority to GBApplication No. GB 1618809.6, filed on Nov. 8, 2016, the entire contentsof each of which being fully incorporated herein by reference.

BACKGROUND OF THE INVENTION 1. Field of the Invention

The field of the invention relates to systems and methods for roboticfruit picking.

A portion of the disclosure of this patent document contains materialthat is subject to copyright protection. The copyright owner has noobjection to the facsimile reproduction by anyone of the patent documentor the patent disclosure, as it appears in the Patent and TrademarkOffice patent file or records, but otherwise reserves all copyrightrights whatsoever.

2. Description of the Prior Art

Horticultural producers depend critically on manual labour forharvesting their crops. Many types of fresh produce are harvestedmanually including berry fruits such as strawberries and raspberries,asparagus, table grapes and eating apples. Manual picking is currentlynecessary because the produce is prone to damage and requires delicatehandling, or because the plant itself is valuable, producing fruitcontinuously over one or more growing seasons. Thus the efficient butdestructive mechanical methods used to harvest crops such as wheat arenot feasible.

Reliance on manual labour creates several problems for producers:

-   -   Recruiting pickers for short, hard picking seasons is risky and        expensive. Domestic supply of picking labour is almost        non-existent and so farmers must recruit from overseas. However,        immigration controls place a large administrative burden on the        producer and increase risk of labour shortage.    -   Supply and demand for low-skilled, migrant labour are        unpredictable because they depend on weather conditions        throughout the growing season and economic circumstances. This        creates significant labour price fluctuations.    -   In extremis, this can lead to crops being left un-harvested in        the field. E.g. a single 250-acre strawberry farm near Hereford        lost more than £200K of produce because of labour shortage in        2007.    -   Human pickers give inconsistent results with direct consequences        for profitability (e.g. punnets containing strawberries with        inconsistent size or shape or showing signs of mishandling would        typically be rejected by customers). Farmers use a variety of        training and monitoring procedures to increase consistency but        these greatly increase cost.

Current technologies for robotic soft fruit harvesting tend to rely onsophisticated hardware and naive robot control systems. In consequence,other soft fruit picking systems have not been commercially successfulbecause they are expensive and require carefully controlledenvironments.

A small number of groups have developed robotic strawberry harvestingtechnology. However, the robots often come at a high cost and still needhuman operators to grade and post-process the fruit. Furthermore, therobots are often not compatible with table top growing systems used inEurope and are too expensive to be competitive with human labour.Expensive hardware and dated object recognition technology are used, andlacked the mechanical flexibility to pick other than carefullypositioned, vertically oriented strawberries or have been poorly suitedto the problem of picking soft fruits that cannot be handled except bytheir stalks. No product offering has therefore materialized.

Hence most of the solutions to date fall down in at least two of thefollowing key areas:

-   -   They require growers to change their working practices        significantly, or don't support the table top growing systems        used in Europe.    -   They rely heavily on human operators. In consequence, they use        large machines with disproportionately high production cost per        unit picking capacity compared to small, autonomous machines        manufactured in larger quantities.    -   They are too expensive to displace human labour at current        prices.

Elsewhere, several academic groups have also applied (mostly quite dated1980's era) robotics and computer vision technology to more generalharvesting applications. However, the resulting systems have been toolimited for commercial exploitation.

Farmers need a dependable system for harvesting their crops, on demand,with consistent quality, and at predictable cost. Such a system willallow farmers to buy a high quality, consistent harvesting capacity inadvance at a predictable price, thereby reducing their exposure tolabour market price fluctuations. The machine will functionautonomously: traversing fields, orchard, or polytunnels; identifyingand locating produce ready for harvest; picking selected crops; andfinally grading, sorting and depositing picked produce into containerssuitable for transfer to cold storage.

SUMMARY OF THE INVENTION

A first aspect of the invention is a robotic fruit picking systemcomprising an autonomous robot that includes the following subsystems:

a positioning subsystem operable to enable autonomous positioning of therobot using a computer implemented guidance system, such as a computervision guidance system;

at least one picking arm;

at least one picking head, or other type of end effector, mounted oneach picking arm to either cut a stem or branch for a specific fruit orbunch of fruits or pluck that fruit or bunch, and then transfer thefruit or bunch;

a computer vision subsystem to analyse images of the fruit to be pickedor stored;

a control subsystem that is programmed with or learns pickingstrategies;

a quality control (QC) subsystem to monitor the quality of fruit thathas been picked or could be picked and grade that fruit according tosize and/or quality; and

a storage subsystem for receiving picked fruit and storing that fruit incontainers for storage or transportation, or in punnets for retail.

We use the term ‘picking head’ to cover any type of end effector; an endeffector is the device, or multiple devices at the end of a robotic armthat interacts with the environment—for example, the head or multipleheads that pick the edible and palatable part of the fruit or that graband cut the stems to fruits.

BRIEF DESCRIPTION OF THE FIGURES

Aspects of the invention will now be described, by way of example(s),with reference to the following Figures, which each show features of theinvention:

FIG. 1 shows a top view (A) and a perspective view (B) of a robotsuitable for fruit picking.

FIG. 2 shows a perspective view of a robot suitable for fruit pickingwith arms in picking position.

FIG. 3 shows an example of a robot suitable for fruit picking.

FIG. 4 shows another example of a robot suitable for fruit picking.

FIG. 5 shows another example of a robot suitable for fruit picking.

FIG. 6 shows an early embodiment of the invention designed for pickingtabletop-grown strawberries.

FIG. 7 shows a system for mounting a ‘vector cable’ to the legs of thetables on which crops are grown using metal brackets that simply clip tothe legs.

FIG. 8 shows a number of line drawings with different views of thepicking arm and picking head alone or in combination.

FIG. 9 shows a sectional view of a typical embodiment of a QC imagingchamber.

FIG. 10 shows a diagram illustrating a space-saving scheme for storingtrays of punnets within a picking robot.

FIG. 11 shows the main components of the end effector.

FIG. 12 shows the hook extended relative to the gripper/cutter.

FIG. 13 shows the hook retracted relative to the gripper/cutter.

FIG. 14 shows the individual parts of the end effector including theblade above the hook.

FIG. 15 shows a diagram illustrating the movement of the hook to effecta capture of the plant stem.

FIG. 16 shows a diagram illustrating the plant stem being captured.

FIG. 17 shows a diagram illustrating the produce being gripped and cutfrom the parent plant.

FIG. 18 shows a diagram illustrating the release operation.

FIG. 19 shows the sequence of operations that constitute the pickingprocess.

FIG. 20 shows the main mechanical constituents of the loop and jawassembly.

FIG. 21 shows the loop and jaw assembly, shown with component 1, theloop, extended.

FIG. 22 shows an exploded diagram of the main constituent parts of theloop and jaw assembly (the loop is omitted for clarity).

FIG. 23 shows the components of the loop actuation mechanism.

FIG. 24 shows the loop/jaw assembly on its approach vector towards thetarget fruit.

FIG. 25 shows the loop extended and the assembly moving in parallel tothe major axis of the target produce and in the direction of thejuncture between stalk and fruit.

FIG. 26 show the loop having travelled past the juncture of the stalkand the fruit, the target produce is now selected and decision step 1(FIG. 19) may be applied.

FIG. 27 shows the loop is retracted to control the position of thetarget produce and the jaw is actuated so as to grab and cut the stalkof the fruit in a scissor-like motion.

FIG. 28 shows the different elements of the cable management system.

FIG. 29 shows drawings of the cable management system in situ within oneof the joints of an arm.

FIG. 30 shows a sequence of drawings with the cable guide rotatingwithin the cable enclosure.

FIG. 31 shows a cutaway view of cable winding.

DETAILED DESCRIPTION

The invention relates to an innovative fruit picking system that usesrobotic picking machines capable of both fully autonomous fruitharvesting and working efficiently in concert with human fruit pickers.

Whilst this description focuses on robotic fruit picking systems, thesystems and methods described can have a more generalized application inother areas, such as robotic litter picking systems.

The picking system is applicable to a variety of different crops thatgrow on plants (like strawberries, tomatoes), bushes (like raspberries,blueberries, grapes), and trees (like apples, pears, logan berries). Inthis document, the term fruit shall include the edible and palatablepart of all fruits, vegetables, and other kinds of produce that arepicked from plants (including e.g. nuts, seeds, vegetables) and plantshall mean all kinds of fruit producing crop (including plants, bushes,trees). For fruits that grow in clusters or bunches (e.g. grapes,blueberries), fruit may refer to the individual fruit or the wholecluster.

Many plants continue to produce fruit throughout the duration of a longpicking season and/or throughout several years of the plant's life.Therefore, a picking robot must not damage either the fruit or the planton which it grows (including any not-yet-picked fruit, whether ripe orunripe). Damage to the plant/bush/tree might occur either as the robotmoves near the plant or during the picking operation.

In contrast to current technologies, our development efforts have beendirectly informed by the needs of real commercial growers. We will avoidhigh cost hardware by capitalizing on state-of-the-art computer visiontechniques to allow us to use lower cost off-the-shelf components. Theappeal of this approach is that the marginal cost of manufacturingsoftware is lower than the marginal cost of manufacturing complexhardware.

An intelligent robot position control system capable of working at highspeed without damaging delicate picked fruit has been developed. Whilsttypical naïve robot control systems are useful for performing repeatedtasks in controlled environments (such as car factories), they cannotdeal with the variability and uncertainty inherent in tasks like fruitpicking. We address this problem using a state-of-the-art reinforcementlearning approach that will allow our robot control system to learn moreefficient picking strategies using experience gained during picking.

Key components of the fruit picking robot are the following:

-   -   A tracked rover capable of navigating autonomously along rows of        crops using a vision-based guidance system.    -   A computer vision system comprising a 3D stereo camera and image        processing software for detecting target fruits, and deciding        whether to pick them and how to pick them.    -   A fast, 6 degree-of-freedom robot arm for positioning a picking        head and camera.    -   A picking head, comprising a means of (i) cutting the strawberry        stalk and (ii) gripping the cut fruit for transfer.    -   A quality control subsystem for grading picked strawberries by        size and quality.    -   A packing subsystem for on-board punnetization of picked fruit.

The picking robot performs several functions completely automatically:

-   -   loading and unloading itself onto and off of a transport        vehicle;    -   navigating amongst fruit producing plants, e.g. along rows of        apple trees or strawberry plants;    -   collaborating with other robots and human pickers to divide        picking work efficiently;    -   determining the position, orientation, and shape of fruit;    -   determining whether fruit is suitable for picking;    -   separating the ripe fruit from the tree;    -   grading the fruit by size and other measures of suitability;    -   transferring the picked fruit to a suitable storage container.

The picking system is innovative in several ways. In what follows, somespecific non-obvious inventive steps are highlighted with wording like“A useful innovation is . . . ”.

1. System Overview

The picking system comprises the following important subsystems:

-   -   Total Positioning Subsystem    -   Picking Arm    -   Picking Head    -   Computer Vision Subsystem    -   Control Subsystem    -   Quality Control (QC) Subsystem    -   Storage Subsystem    -   Mapping Subsystem    -   Management Subsystem

The main purpose of the robot Total Positioning System is physically tomove the whole robot along the ground. When the robot is within reach oftarget fruit, the Picking Arm moves an attached camera to allow theComputer Vision Subsystem to locate target fruits and determine theirpose and suitability for picking. The Picking Arm also positions thePicking Head for picking and moves picked fruit to the QC Subsystem (andpossibly the Storage Subsystem). The Total Positioning System and thePicking Arm operate under the control of the Control Subsystem, whichuses input from the Computer Vision Subsystem to decide where and whento move the robot. The main purpose of the Picking Head is to cut thefruit from the plant and to grip it securely for transfer to the QC andStorage subsystems. Finally, the QC Subsystem is responsible for gradingpicked fruit, determining its suitability for retail or other use, anddiscarding unusable fruit.

FIG. 1 shows a top view (a) and a perspective view (b) of a robotsuitable for fruit picking. The robot includes a Positioning Subsystemoperable to enable autonomous positioning of the robot using a computerimplemented guidance system, such as a Computer Vision Guidance System.Two Picking Arms (100) are shown for this configuration. A Picking Head(101) is mounted on each Picking Arm (100) to either cut a stem orbranch for a specific fruit or bunch of fruits or pluck that fruit orbunch, and then transfer the fruit or bunch. The Picking Head alsoincludes the camera component (102) of the Computer Vision Subsystem,which is responsible for analysing images of the fruit to be picked orstored. A Control Subsystem is programmed with or learns pickingstrategies. A Quality Control (QC) Subsystem (103) monitors the qualityof fruit that has been picked or could be picked and grade that fruitaccording to size and/or quality and a Storage Subsystem (104) receivesthe picked fruit and stores that fruit in containers for storage ortransportation, or in punnets for retail. FIG. 2 shows a perspectiveview of the robot with arms in picking position.

FIGS. 3-5 show examples of a robot suitable for fruit picking. Threedifferent conceptual illustrations show the robot configured in severaldifferent ways, suitable for different picking applications. Importantsystem components shown include the tracked rover, two Picking Arms, andassociated Quality Control units (QC). Multiple trays are used to storepunnets of picked fruit and are positioned so that a human operator caneasily remove full trays and replace them with empty trays. A discardshoot is positioned adjacent to the Quality Control units for fruit notsuitable for sale. When the robot picks rotten or otherwise unsuitablefruit (either by accident or design), it is usually desirable to discardthe rotten fruit into a suitable container within the robot or onto theground. A useful time-saving innovation is to make the containeraccessible via a discard chute with its aperture positioned at thebottom of the QC rig so that the arm can drop the fruit immediatelywithout the need to move to an alternative container. A relatedinnovation is to induce positive or negative air pressure in the chuteor the body of the imaging chamber (e.g. using a fan) to ensure thatfungal spores coming from previously discarded fruit are kept away fromhealthy fruit in the imaging chamber. The two Picking Arms can bepositioned asymmetrically (as shown in FIG. 5) to increase reach at theexpense of picking speed.

FIG. 6 shows an early embodiment of the invention designed for pickingtabletop-grown strawberries and having a single Picking Arm and twostorage trays.

These subsystems will be described in more detail in the followingsections.

2. Total Positioning Subsystem

The Total Positioning Subsystem is responsible for movement of the wholerobot across the ground (typically in an intendedly straight linebetween a current position and an input target position). The TotalPositioning Subsystem is used by the Management Subsystem to move therobot around. The Total Positioning System comprises a means ofdetermining the present position and orientation of the robot, a meansof effecting the motion of the robot along the ground, and a controlsystem that translates information about the current position andorientation of the robot into motor control signals.

2.1. Pose Determination Component

The purpose of the pose determination component is to allow the robot todetermine its current position and orientation in a map coordinatesystem for input to the Control Component. Coarse position estimates maybe obtained using differential GPS but these are insufficiently accuratefor following rows of crops without collision. Therefore, a combinationof additional sensors is used for more precise determination of headingalong the row and lateral distance from the row. The combination mayinclude ultrasound sensors for approximate determination of distancefrom the row, a magnetic compass for determining heading,accelerometers, and a forwards or backwards facing camera fordetermining orientation with respect to the crop rows. Information fromthe sensors is fused with information from the GPS positioning system toobtain a more accurate estimate than could be obtained by either GPS orthe sensors individually.

An innovative means of allowing the robot to estimate its position andorientation with respect to a crop row is to measure its displacementrelative to a tensioned cable (perhaps of nylon or other low costmaterial) that runs along the row (a ‘vector cable’).

One innovative means of measuring the displacement of the robot relativeto the vector cable is to use a computer vision system to measure theprojected position of the cable in 2D images obtained by a cameramounted with known position and orientation in the robot coordinatesystem. As a simplistic illustration, the orientation of a horizontalcable in an image obtained by a vertically oriented camera has a simplelinear relationship with the orientation of the robot. In general, theedges of the cable will project to a pair of lines in the image, whichcan be found easily by standard image processing techniques, e.g. byapplying an edge detector and computing a Hough transform to find longstraight edges. The image position of these lines is a function of thediameter of the cable, the pose of the camera relative to the cable, andthe camera's intrinsic parameters (which may be determined in advance).The pose of the camera relative to the cable may then be determinedusing standard optimization techniques and an initialization provided bythe assumption that the robot is approximately aligned with the row. Aremaining one-parameter ambiguity (corresponding to rotation of thecamera about the axis of the cable) may be eliminated knowing theapproximate height of the camera above the ground.

Another innovative approach to determining position relative to thevector cable is to use a follower arm (or follower arms). This isconnected at one end to the robot chassis by means of a hinged joint andat the other to a truck that runs along the cable. The angle at thehinged joint (which can be measured e.g. using the resistance of apotentiometer that rotates with the hinge) can be used to determine thedisplacement relative to the cable. Two follower arms (e.g. one at thefront and one at the back) is sufficient to determine displacement andorientation.

A related innovation is a bracket that allows vector cables to beattached easily to the legs of the tables on which crops such asstrawberries are commonly grown. This is illustrated in FIG. 7. Thebrackets allow the cable to be positioned a small and consistentdistance above the ground, typically 20 cm, so that is easy for humansto step over it whilst ducking underneath elevated tables to move fromrow to row. One limitation of this approach is that the robot mayreceive spurious position information if the follower arm falls off thecable. Therefore, a useful additional innovation is to equip the truckwith a microswitch positioned so as to break an electrical circuit whenthe truck loses contact with the cable. This can be used to allow thecontrol software to detect this failure condition and stop the robot(typically, the control software waits until the duration of detectedloss of contact between the truck and the cable is greater than sometime threshold to eliminate false detections due to bouncing of thetruck on the cable). Since the follower arm might be subjected tosignificant forces in the event of collision or other kind of failure ofcontrol system failure, another useful innovation is to use a magneticcoupling to attach an outer portion of the follower arm to an innerportion. Then in the event of failure the parts of the follower arm canseparate without suffering permanent damage. The magnetic coupling caninclude electrical connections required to complete an electricalcircuit (typically the same circuit that is interrupted by a microswitchin the truck). By this means separation of the follower arm can alsotrigger the control software to stop the robot. Another benefit of themagnetic coupling arranging is ease of attachment of the follower arm bya human supervisor.

An innovative aspect of the pose determination component is a computervision based system for determining the heading and lateral position ofa robot with respect to a row of crops using images obtained by aforwards or backwards facing camera pointing approximately along therow. Such a system can be used to drive the robot in the middle of twocrop rows or at a fixed distance from a single crop row. In oneembodiment of this idea, this is achieved by training a regressionfunction implemented using a convolutional neural network (or otherwise)to predict robot heading and lateral position with respect to rows ofcrops as a function of an input image. Training data may be obtained bydriving a robot equipped with multiple forwards and/or backwards facingcameras between representative crop rows under human remote control.Because the human controller keeps the robot approximately centredbetween the rows (with heading parallel to the rows), each frame can beassociated with approximate ground truth heading and lateraldisplacement information. Multiple cameras are used to provide atraining images corresponding to different approximate lateraldisplacements from the row. Training images corresponding to differentrobot headings can be obtained by resampling images obtained by aforwards looking camera using an appropriate 3-by-3 homography (whichcan be computed trivially from known camera intrinsic calibrationparameters).

A related innovation is to obtain additional training image data atnight using an infrared illuminator and suitable infrared receptivecameras.

In another embodiment of this idea, a computer vision system is designedto detect (in images obtained by a forwards or backwards facing camera)the vertical legs of the tables on which crops are grown. Verticallyoriented table legs define vertical lines in the world, which project tolines in the perspective view. Under the assumption that the legs ofeach table are evenly spaced, vertically oriented, and arranged in astraight line, projected image lines corresponding to a sequence ofthree or more table legs are sufficient to determine the orientation ofa calibrated camera and its lateral displacement with respect to a 3Dcoordinate system defined by the legs.

FIG. 7 shows a system for mounting a ‘vector cable’ to the legs of thetables on which crops are grown (a) using metal brackets (b) that simplyclip to the legs. The cable (e.g. a nylon rope) may be secured to themetal brackets using n-shaped metal spring clips or tape (not shown).The row follower arm (seen in b) comprises a truck that rests on thecable and an arm that attaches the truck to the robot (not shown). Thebrackets are shaped such that the truck is not impeded as it traversesthem.

2.2 Motor Control Component

The purpose of the motor control component is to map pose informationprovided by the pose determination component to motor control signals tomove the robot in a given direction. It supports two kinds of motion:(i) moving a given distance along a row of plants and (ii) moving to agiven point by travelling in an intendedly straight line. The motorcontrol system uses a PID controller to map control inputs obtained fromthe pose determination component to motor control signals.

2.3. Rover

An important component of the Total Positioning System is the rover, themeans by which the robot moves over the ground. Typically, movement overthe ground is achieved using powered wheels with tracks. A usefulinnovation is a mechanism to allow the tracks to be removed so that therobot can also run on rails.

3. Picking Arm

The Picking Arm is a robot arm with several (typically 6) degrees offreedom that is mounted to the main body of the robot. Whereas thepurpose of the Total Positioning System is to move the whole robot alongthe ground, the purpose of the Picking Arm is to move the Picking Head(and its computer vision camera) to appropriate positions for locating,localizing, and picking target fruit. Once it is in the pickingposition, the Picking Head executes a picking routine that comprises asequence of mechanical actions including separation, gripping, andcutting (see the Picking Head description below). Picking positions arechosen by the Control Subsystem to maximize picking performanceaccording to a desired metric.

Before the Picking Head can be positioned to pick a target fruit theComputer Vision Subsystem must carry out several important operations:(i) detecting the target fruit, (ii) detecting obstructions that mightcomplicate picking of the target fruit (e.g. leaves), (iii) determiningthe pose and shape of the target fruit. So that the Computer VisionSubsystem can perform these functions, the Picking Arm may be equippedwith a monocular or stereo camera, mounted e.g. to the end of the arm.The benefit of having a camera mounted to the arm is the possibility ofmoving the camera to find viewpoints free from sources of occlusion thatwould otherwise prevent reliable detection or localization of the targetfruit (leaves, other fruits, etc.).

Finally, the Picking Arm must move the Picking Head to an appropriatepose for picking without colliding with the plant, or the supportinfrastructure used to grow the plant, or itself. This is achieved usinga route-planning algorithm described in the Control Subsystem sectionbelow.

FIG. 8 shows a number of line drawings with different views of thePicking Arm and Picking Head alone or in combination. The Picking Headshown includes a hook and camera system.

4. Picking Head

The purpose of the Picking Head is to sever the target fruit from theplant, to grasp it securely while it is moved to the QC and StorageSubsystems, and to release it. A secondary purpose is to move leaves andother sources of occlusion out of the way so fruit can be detected andlocalized, and to separate target fruit from the plant (before it ispermanently severed) to facilitate determination of picking suitability.

Picking soft fruits like strawberries is challenging because physicalhandling of the fruit can cause bruising, reducing saleability.Therefore, such fruits are ideally picked by severing the stem withouthandling the body of the fruit. An inventive aspect of our system is theuse of a stem-severing Picking Head that works in three phases(‘grab-grip-cut’):

-   -   1. Physical separation of the fruit from the remainder of the        plant (‘grab’). In advance of permanently severing the fruit        from the plant, this step facilitates (i) deciding whether or        not the fruit is suitable for picking and (ii) increases the        chances that picking in step 2 will proceed successfully without        damage to the target fruit of the rest of the plant.    -   2. Gripping of the picked fruit by its stem (‘grip’).    -   3. Severing the stem (above the point at which it is gripped) so        as permanently to separate the fruit from the plant (‘cut’).

The introduction of the physical separation phase (which take placebefore the fruit is permanently severed from the plant) confers severalbenefits. Since target fruit may be occluded by leaves or other fruit,pulling it further away from the plan facilitates a better view,allowing the computer vision system to determine more reliably whetherthe fruit is ready for picking and whether the picking procedure islikely to be successful (for example because other fruits are in thetarget vicinity). A related innovation is a mechanical gripper that canrotate the gripped fruit during this before-picking inspection phase,e.g. by applying a twisting force to its stalk or otherwise. By thismeans, a camera or other sensors can obtain information about parts ofthe fruit that would not otherwise have been visible. One benefit ofthis innovation it the possibility of deciding to postpone picking afruit that appears unripe on the once-hidden side.

A possible further innovation is to combine the grip and cut phases (2and 3) by means of exploiting the gripping action to pull the stemagainst a cutting blade or blades.

Appendix A describes several innovative mechanical Picking Head designsembodying some of these ideas.

For some soft fruits such as raspberries, it is necessary to remove thefruit from its stem during picking. For such fruits, a useful innovationis to pick the fruit by first severing and gripping its stem and then toremove the body of the fruit from its stem in a subsequent operation.Compared to picking techniques that require holding or gripping the bodyof the fruit, important benefits if this approach include: (i)minimizing handling of the body of the fruit (which can significantlyreduce shelf life, e.g. due to the transference of disease-causingpathogens from fruit to fruit on the surface of the handling device),(ii) the possibility of imaging the body of the picked fruit from alldirections for quality control, and (iii) the possibility of removingthe stem under controlled conditions.

Various means of removing the picked fruit from the stem are possible.One innovative approach is to pull the fruit from its stem using a jetof compressed air. This allows contact forces to be distributed evenlyover a large contact area, minimizing bruising. Another possibility isto pull the fruit by it stem through a collar, shaped to facilitateforcing the body of the fruit off the stem. Depending on the specifictype of fruit, collars might be designed to either to distribute thecontact force over a large area of the body of the fruit or toconcentrate the contact force (possibly via a cutting edge resemblingthat of a knife of row of needles) in a circular pattern surrounding thestem. A related innovation is to clean the collar after each use or toprovide the collar with a disposable surface to reduce the likelihood oftransfer of pathogens from fruit to fruit.

Another innovation is to use the inertia of the body of the fruit toseparate the body of the fruit from the receptacle. This might beachieved by holding the fruit by its stalk and rotating it about an axisperpendicular to its stalk at a high enough angular velocity. Theadvantage of this approach is that inertial forces effectively act overthe entire mass of the body of the fruit, eliminating the need forcontact forces at the surface (more likely to cause bruising becausethey are applied over a small contact area, increasing localizedpressure). One limitation of this approach is that, when the body of thefruit separates from the receptacle, it will fly off at speed on atangent to the circle of rotation, necessitating some means of arrestingits motion sufficiently slowly that it doesn't suffer bruising.Therefore, another innovation is to use a reciprocating back-and-forthmotion of the fruit or its stalk in the direction approximatelyperpendicular to the stalk or an oscillatory rotary motion with an axisof rotation approximately parallel to the stalk. By performing themotion at an appropriate frequency and with appropriate amplitude it ispossible reliably to separate the body of the fruit from the huskwithout causing it to fly off at high velocity.

After picking, the Picking Head grips the fruit as it transferred by thePicking Arm to the Quality Control (QC) Subsystem. In a simpleembodiment, the Picking Arm itself might be used to position the pickedfruit inside the imaging component of the QC Subsystem beforesubsequently moving the fruit to the storage. However, time spenttransferring picked fruit to the QC/Storage Subsystem is unproductivebecause the arm is not being used for picking during the transfer.Therefore, a useful innovation is to include multiple picking units on asingle multiplexed Picking Head. This means that several fruits in aparticular local vicinity can be picked in succession before the arm hasto perform the time-consuming movement between the plant and theQC/Storage components and back. This means that the transfer overheadcan be amortized over more picked fruits, increasing productivity. Thisis particularly advantageous in the common case that fruit are bunchedon the plant/tree such the robot arm needs to move only a small distanceto pick several targets before transfer.

Picking units on the multiplexed Picking Head must be arranged so thatinactive picking units do not interfere with the operation of activepicking units, or collide with the arm, or other objects. Innovativeways of achieving this include:

-   -   mounting the picking units radially about an axis chosen so that        inactive picking units are oriented away from the active picking        unit and the fruit being picked.    -   making each picking unit extend independently so it can engage        with the fruit while others do not disturb the scene.

Picking units typically have several moving parts e.g. for hooking,cutting, etc., which may need to be driven independently. However, whenmultiplexing multiple units on a single Picking Head, if each movingpart is driven with its own actuator, the arm payload increasesproportionally to the number of picking units, which would adverselyaffect arm speed, accuracy, and the overall cost of the machine. Severalinnovative aspects of the implementation of the multiplexed Picking Headkeep the overall mass of the multiplexed Picking Head low to allow thearm to move quickly and accurately:

-   -   Multiple picking functions on a picking unit can be driven by a        single actuator or motor, selectively engaged by lightweight        means, for example electromagnets; an engaging pin; rotary tab;        or similar. This is challenging as the different functions may        require different actuator characteristics    -   A single motor or actuator can drive one function across all        units on the head, selectively engaged by means of an        electromagnet; an engaging pin; rotary tab; or similar. This is        reasonably straightforward.    -   The functions can be driven by lightweight means from elsewhere        in the system, for example using a Bowden cable, torsional drive        cable/spring, pneumatic or hydraulic means.

5. Computer Vision Subsystem

The purpose of the Computer Vision Subsystem is to locate target fruits,determine their pose in a robot coordinate system, and determine whetherthey are suitable for picking, i.e. before the fruit is permanentlyseparated from the plant.

To achieve this, the Computer Vision Subsystem uses one or more camerasmounted to the end of the movable Picking Arm (or in general to anyother part of the robot such as the chassis). The camera attached to therobot arm is moved under computer control to facilitate the detection oftarget fruits, estimation of their pose (i.e. position and orientation),and determination of their likely suitability for picking. Poseestimates and picking suitability indicators associated with each targetfruit may be refined progressively as the arm moves. However, thisrefinement stage takes time, which increases the time required forpicking. Therefore, an important innovation is a scheme for moving thearm efficiently to optimize trade-off between picking speed and pickingaccuracy (this scheme is described in more detail in the Robot ControlSubsystem section, below).

The Computer Vision Subsystem operates under the control of the RobotControl Subsystem, which makes a continuous sequence of decisions aboutwhich action to perform next, e.g. moving the arm-mounted camera to newviewpoints to facilitate discovery of more target fruits, or moving thecamera to new points in the local vicinity of a target fruit so as torefine estimates of its position/orientation or indicators of pickingsuitability.

In outline, the Computer Vision Subsystem works as follows:

-   -   1. Under the control of the Robot Control Subsystem, the camera        captures images of the scene from multiple viewpoints.    -   2. Target fruits are detected in the captured images by        pixel-wise semantic segmentation.    -   3. Approximate estimates of pose and shape are recovered for        each detected fruit.    -   4. More accurate estimates of pose and shape are recovered by        combining information in multiple views with statistical prior        knowledge. This is achieved by adapting the parameters of a        generative model of strawberry appearance to maximize agreement        between the predictions and the images.    -   5. Picking success probability for each detected fruit is        estimated from visual and geometric cues.    -   6. Under the control of the Control Subsystem, additional images        of a particular target fruit may be captured from new viewpoints        so as to increase picking success probability.

The important steps are described in more detail below.

Image Capture.

An important challenge is to control the exposure of the camera systemto obtain images that are consistently correctly exposed. Failure to dothis increases the amount of variability in the images, compromising theability of the machine-learning-based target detection software todetect fruit accurately and reliably. One exposure control strategy isto obtain an image of a grey card exposed to ambient lightingconditions. This image is then analysed to determine the adjustments toexposure time and/or colour channel gains required to ensure that thegrey card appears with a predetermined target colour value. A grey cardmight be positioned on the robot chassis with reach of the Picking Armsand oriented horizontally to measure ambient illumination arriving fromthe approximate direction of the sky. However, a potential limitation ofthis approach is that the illumination of the grey card may not berepresentative of the illumination of the plant or target fruit.Therefore, in a system where a (stereo) camera is incorporated withinthe Picking Head, a useful innovation is to arrange that a part of thePicking Head itself can be used as an exposure control target. Aprerequisite is that the exposure control target must appear within thefield of view of the camera. A suitable target could be a grey cardimaged from in front or a translucent plastic diffuser imaged fromunderneath.

Real world lighting conditions can compromise image quality, limitingthe effectiveness of image processing operations such as target fruitdetection. For example, images obtained by a camera oriented directlytowards the sun on a cloudless day may exhibit lens flare. Therefore, auseful innovation is to have control system software use the weatherforecast to schedule picking operations so that the robot's camerasystems are oriented to maximize the quality of the image data beingobtained as a function of expected lighting conditions over a givenperiod. For example, on a day that is forecast to be sunny in a farmwhere fruit is grown in rows, the robot might pick on one side of therows in the morning and the other side of the rows in the afternoon. Ona cloudy day, robots might more usefully pick on both sides of the rowsimultaneously to amortize the cost of advancing the robot along the rowover more picked fruit at each position. A related innovation is toadapt viewpoints dynamically as a function of lighting conditions tomaximize picking performance. For example, the Picking Head might beangled downwards in conditions of direct sunlight to avoid lens flareeven at the expense of a reduction in working volume.

Target Detection.

Target fruit is detected automatically in images obtained by a cameramounted to the Picking Arm or elsewhere. A machine learning approach isused to train a detection algorithm to identify fruit in RGB colourimages (and/or in depth images obtained by dense stereo or otherwise).To provide training data, images obtained from representative viewpointsare annotated manually with the position and/or extent of target fruit.Various embodiments of this idea are possible:

-   -   1. A decision forest classifier or convolutional neural network        (CNN) may be trained to perform semantic segmentation, i.e. to        label pixels corresponding to ripe fruit, unripe fruit, and        other objects. Pixel-wise labelling may be noisy, and evidence        may be aggregated across multiple pixels by using a clustering        algorithm.    -   2. A CNN can be trained to distinguish image patches that        contain a target fruit at their centre from image patches that        do not. A sliding window approach may be used to determine the        positions of all image patches likely to contain target fruits.        Alternatively, the semantic labelling algorithm 1 may be used to        identify the likely image locations of target fruits for        subsequent more accurate classification by a (typically more        computationally expensive) CNN.

Target Pose Determination.

Picking Heads for different types of fruit may work in different ways,e.g. by cutting the stalk or by twisting the fruit until the stalk issevered (see above, and Appendix A). Depending on the Picking Headdesign, picking a target fruit may necessitate first estimating theposition and orientation (or pose) of the fruit or its stalk (in whatfollows, fruit should be interpreted to mean the body of the fruit orits stalk or both). Rigid body pose in general has 6 degrees of freedom(e.g. the X, Y, Z coordinates of a fruit in a suitable world coordinatesystem and the three angles describing its orientation relative to theworld coordinate system's axes). Pose may be modelled as a 4-by-4homography that maps homogenous 3D points in a suitable fruit coordinatesystem into the world coordinate system. The fruit coordinate system canbe aligned with fruits of specific types as convenient. For example, theorigin of the coordinate system may be located at the point ofintersection of the body of the fruit and its stalk and the first axispoints in the direction of the stalk. Many types of fruit (such asstrawberries and apples) and most kinds of stalk have a shape with anaxis of approximate rotational symmetry. This means that 5 degrees offreedom typically provide a sufficiently complete representation of posefor picking purposes, i.e. the second and third axes of the fruitcoordinate system can be oriented arbitrarily.

The robot determines the pose of target fruit using images obtained frommultiple viewpoints, e.g. using a stereo camera or a monocular cameramounted to the moving Picking Arm. For example, the detected position ofa target fruit in two or more calibrated views is sufficient toapproximately to determine its X,Y,Z position by triangulation. Theorientation of the fruit or its stalk may then be estimated byassumption (e.g. the assumption that fruits hang vertically) orrecovered from image data.

A useful innovation is to use a learned regression function to mapimages of target fruits directly to their orientation in a cameracoordinate system. This can be achieved using a machine learningapproach whereby a suitable regression model is trained to predict thetwo angles describing the orientation of an approximately rotationallysymmetric fruit from images (including monocular, stereo, and depthimages). This approach is effective for fruits such as strawberries thathave surface texture that is aligned with the dominant axis of thefruit. Suitable training images may be obtained using a camera mountedto the end of a robot arm. First, the arm is moved manually until thecamera is approximately aligned with a suitable fruit-based coordinatesystem and a fixed distance away from the fruit's centroid. The arm isaligned so the fruit has canonical orientation in a camera image, i.e.so that the two or three angles used to describe orientation in thecamera coordinate frame are 0. Then the arm moves automatically toobtain additional training images from new viewpoints with different,known relative orientations of the fruit. Sufficiently high qualitytraining data can be obtained by having a human operator judge alignmentbetween the camera and the fruit coordinate system visually byinspection of the scene and the video signal produced by the camera.Typically training images are cropped so that the centroid of thedetected fruit appears in the centre of the frame and scaled so that thefruit occupies constant size. Then a convolutional neural network orother regression model is trained to predict fruit orientation inpreviously unseen images. Various image features are informative as tothe orientation of the fruit in the camera image frame (and can beexploited automatically by a suitable machine learning approach), e.g.the density and orientation of any seeds on the surface of the fruit,the location of the calyx (the leafy part around the stem), and imagelocation of the stalk.

Because knowledge of the orientation of the stalk may be very importantfor picking some types of fruits (or otherwise informative as to theorientation of the body of the fruit), another useful innovation is astalk detection algorithm that identifies and delineates stalks inimages. A stalk detector can be implemented by training a pixel-wisesemantic labelling engine (e.g. a decision forest or CNN) using manuallyannotated training images to identify pixels that lie on the centralaxis of a stalk. Then a line growing algorithm can be used to delineatevisible portions of stalk. If stereo or depth images are used, thenstalk orientation can be determined in a 3D coordinate frame by matchingcorresponding lines corresponding to a single stalk in two or moreframes. Solution dense stereo matching problem is considerablyfacilitated by conducting semantic segmentation of the scene first(stalks, target fruits). Assumptions about the range of depths likely tobe occupied by the stalk can be used to constrain the stereo matchingproblem.

Given an approximate pose estimate for a target fruit, it may be thatobtaining an additional view will improve the pose estimate, for exampleby revealing an informative part of the fruit such as the point wherethe stalk attaches. Therefore, a useful innovation is an algorithm forpredicting the extent to which additional views out of a set ofavailable viewpoints will most significantly improve the quality of aninitial pose estimate. Pose estimates obtained using multiple views andstatistical prior knowledge about the likely shape and pose of targetfruits can be fused using an innovative model fitting approach (seebelow).

Size and Shape Determination and Pose Estimate Refinement.

Whether a target fruit is suitable for picking may depend on its shapeand size, e.g. because a customer wants fruit with diameter in aspecified range. Furthermore, certain parameters of the picking systemmay need to be tuned considering the shape and size of the fruit, e.g.the trajectory of the Picking Head relative to the fruit during theinitial ‘grab’ phase of the picking motion (see above). Therefore, itmay be beneficial to estimate the shape and size of candidate fruitsbefore picking as well as to refine (possibly coarse) pose estimatesdetermined as above. This can be achieved using images of the fruit(including stereo images) obtained from one or more viewpoints.

An innovative approach to recovering the 3D shape of a candidate fruitfrom one or more images is to adapt the parameters of a generative modelof the fruit's image appearance to maximize the agreement between theimages and the model's predictions, e.g. by using Gauss-Newtonoptimization. This approach can also be used to refine a coarse initialestimate of the fruit's position and orientation (provided as describedabove). A suitable model could take the form of a (possibly textured)triangulated 3D mesh projected into some perspective views. The shape ofthe 3D mesh could be determined by a mathematical function of someparameters describing the shape of the fruit. A suitable function couldbe constructed by obtaining 3D models of a large number of fruits, andthen using Principal Component Analysis (or other dimensionalityreduction strategy) to discover a low-dimensional parameterization ofthe fruit's geometry. Another simpler but effective approach is to handcraft such a model, for example by assuming that the 3D shape of fruitcan be explained as a volume of revolution to which parametricanisotropic scaling has been applied in the plane perpendicular to theaxis. A suitable initialization for optimization can be obtained byusing the 2D image shape (or the mean 2D image shape of the fruit) todefine a volume of revolution. The pose parameters can be initializedusing the method described above.

A key benefit of the model fitting approach is the possibility ofcombining information from several viewpoints simultaneously. Agreementbetween the real and predicted image might be measured, e.g. using thedistance between the real and predicted silhouette or, for a model thatincludes lighting or texture, as the sum of squared differences betweenpixel intensity values. A useful innovation is to use the geometricmodel to predict not only the surface appearance of the fruit but theshadows cast by the fruit onto itself under different, controlledlighting conditions. Controlled lighting can be provided by one or moreilluminators attached to the end of the robot arm. Another usefulinnovation is to model agreement using a composite cost function thatcomprises terms reflecting agreement both between silhouette and stalk.

Another benefit of the model fitting approach is the possibility ofcombining image evidence with statistical prior knowledge to obtain amaximum likelihood estimate of shape and pose parameters. Statisticalprior knowledge can be incorporated by penalizing unlikely parameterconfigurations that are unlikely according to a probabilistic model. Onevaluable innovation is the use for this purpose of a statistical priorthat model the way that massive fruits hang from their stalks under theinfluence of gravity. In a simple embodiment, the prior might reflectour knowledge that fruits (particularly large fruits) tend to hangvertically downwards from their stalks. Such a prior might take thesimple form of a probability distribution over fruit orientation. A morecomplex embodiment might take the form of the joint distribution overthe shape and size of the fruit, the pose of the fruit, and the shape ofthe stalk near the point of attachment to the fruit. Suitableprobability distributions are usually formed by making geometricmeasurements of fruit growing under representative conditions.

Some Picking Head designs make it possible physically to separate acandidate fruit further from the plant and other fruits in the bunchbefore picking (see above). For example, the ‘hook’ design of PickingHead (see Appendix A) allows a candidate fruit to be supported by itsstalk so that it hangs at a predictable distance from a camera mountedto the robot arm. One benefit of this innovation is the possibility ofcapturing an image (or stereo image) of the fruit from a controlledviewpoint, thereby facilitating more accurate determination of the sizeand shape, e.g. via shape from silhouette.

Determination of Picking Suitability.

An attempt to pick a target fruit might or might not be successful.Successful picking usually means that (i) the picked fruit is suitablefor sale (e.g. ripe and undamaged) and delivered to the storagecontainer in that condition, (ii) no other part of the plant or growinginfrastructure is damaged during picking, and (iii) the Picking Arm doesnot undergo any collisions that could interfere with its continuingoperation. However, in the case of rotten fruits that are picked anddiscarded to prolong the life of the plant, it is not a requirement thatthe picked fruit is in saleable condition.

A valuable innovation is to determine the picking suitability of atarget fruit by estimating the statistical probability that an attemptto pick it will be successful. This probability can be estimated beforeattempting to pick a the target fruit via a particular approachtrajectory and therefore can be used by the Control Subsystem to decidewhich fruit to pick next and how to pick it. For example, the fruitsthat are easiest to pick (i.e. those most likely to be pickedsuccessfully) might be picked first to facilitate subsequent picking offruits that are harder to pick, e.g. because they are partly hiddenbehind other fruits. The picking success probability estimate can alsobe used to decide not to pick a particular target fruit, e.g. becausethe expected cost of picking in terms of damage to the plant or pickedfruit will not outweigh the benefit provided by having one more pickedfruit. The Control Subsystem is responsible for optimizing pickingschedule to achieve the optimal trade-off between picking speed andfailure rate (see below).

An important innovation is a scheme for estimating the probability ofpicking success using images of the scene obtained from viewpoints nearthe target fruit. For example, we might image the surface of the fruitby moving a camera (possibly a stereo or depth camera) mounted to thePicking Arm's end effector in its local vicinity. Various imagemeasurements might be used as indicators of picking success probability,including e.g. (i) the estimated pose and shape of the fruit and itsstalk, (ii) the uncertainty associated with the recovered pose and shapeestimates, (iii) the colour of the target fruit's surface, (iv) theproximity of detected obstacles, and (v) the range of viewpoints fromwhich the candidate fruit is visible.

A suitable statistical model for estimating picking success probabilitymight take the form of a multivariate histogram or Gaussian defined onthe space of all picking success indicators. An important innovation isto learn and refine the parameters of such a model using picking successdata obtained by working robots. Because the Quality Control Subsystemprovides accurate judgments about the saleability of picked fruits, itsoutput can be used as an indicator of ground truth picking success orfailure. An online learning approach can be used to update the modeldynamically as more data are generated to facilitate rapid adaptation ofpicking behaviour to the requirements of a new farm or phase of thegrowing season. Multiple robots can share and update the same model.

Since the Picking Head might approach a target fruit via a range ofpossible trajectories (depending on obstacle geometry and the degrees offreedom of the Picking Arm), the probability of picking success ismodelled as a function of hypothesized approach trajectory. By thismeans, the Control Subsystem can decide how to pick the fruit to achievethe best trade-off between picking time and probability of pickingsuccess. The probability of collision between the Picking Arm and thescene can be modelled during the path planning operation using anexplicit 3D model of the scene (as described in the Control Subsystemsection below). However, an alternative and innovative approach is touse an implicit 3D model of the scene formed by the range of viewpointsfrom which the target fruit can be observed without occlusion. Theunderlying insight is that if the target fruit is wholly visible from aparticular viewpoint, then the volume defined by the inverse projectionof the 2D image perimeter of the fruit must be empty between the cameraand the fruit. By identifying one or more viewpoints from which thetarget fruit appears un-occluded, obstacle free region of space isfound. Provided no part of the Picking Head or Arm strays outside ofthis region of space during picking, there should be no collision.Occlusion of the target fruit by an obstacle between the fruit and thecamera when viewed from a particular viewpoint can be detected byseveral means including e.g. stereo matching.

Another important innovation is a Picking Head that can pull the targetfruit away from the plant, to facilitate more reliable determination ofthe fruit's suitability for picking before the fruit is permanentlysevered from the plant. Novel Picking Head designs are described inAppendix A.

6. Quality Control Subsystem

The primary function of the Quality Control Subsystem is to assign ameasure of quality to individual picked fruits (or possibly individualbunches of picked fruits for fruits that are picked in bunches).Depending on the type of fruit being picked and the intended customer,quality is a function of several properties of the fruit, such asripeness, colour, hardness, symmetry, size, stem length. Picked fruitmay be assigned a grade classification that reflects its quality, e.g.grade 1 (symmetric) or grade 2 (shows significant surface creasing) orgrade 3 (very deformed or otherwise unsuitable for sale). Fruit of toolow quality for retail sale may be discarded of stored separately foruse in other applications, e.g. jam manufacture. An importantimplementation challenge is to ensure that the QC step can be carriedout quickly to maximize the productivity of the picking robot.

A secondary function of the QC Subsystem is to determine a more accurateestimate of the fruit's size and shape. This estimate may be used forseveral purposes, e.g.

-   -   for quality grading, since any asymmetry in the 3D shape for the        fruit may be considered reason to assign a lower quality grade;    -   as a means of estimating the fruit's mass and thereby of        ensuring that the require mass of fruit is placed in each punnet        according to the requirements of the intended customer for        average or minimum mass per punnet;    -   to facilitate more precise placement of the fruit in the storage        container, and therefore to minimize the risk of bruising due to        collisions.

The QC Subsystem generates a quality measure for each picked piece offruit by means of a computer vision component comprising some cameras,some lights, and some software for image capture and analysis.Typically, the cameras are arranged so as to obtain images of the entiresurface of a picked fruit that has been suitably positioned, e.g. by thePicking Arm. For example, for fruits like strawberries, which can beheld so as to hang vertically downwards from their stalks, one cameramight be positioned below the fruit looking upwards and several morecameras might be positioned radially about a vertical axis lookinginwards. However, one limitation of this scheme is that a large amountof volume would be required to accommodate cameras (allowing for thecamera-object distance, the maximum size of the fruit, and the toleranceinherent in the positioning of the fruit). One solution might be torotate the fruit in front of a single camera to obtain multipleviews—however any undamped motion of the fruit subsequent to rotationmight complicate imaging. Therefore, another useful innovation is to usemirrors positioned and oriented so as to provide multiple virtual viewsof the fruit to a single camera mounted underneath the fruit. Thisscheme considerably reduces the both the cost and the size of the QCSubsystem. Cameras and/or mirrors are typically arranged so that thefruit appears against a plain background in all views to facilitatesegmentation of the fruit in the images.

Another useful innovation is to obtain multiple images under differentlighting conditions. For example, this might be achieved by arranging aseries of LED lights in a circle around the fruit and activating themone at a time, capturing one exposure per light. This innovationconsiderably increases the informativeness of the images becausedirectional lights induce shadows both on the surface of the fruit andon a suitability positioned background screen. Such shadows can be usedto obtain more information about the 3D shape of the fruit and e.g. thepositions of any surface folds that could reduce saleability.

Using these images, image analysis software measures the fruit's 3Dshape and detects various kinds of defect (e.g. rot, bird damage, sprayresidue, bruising, mildew, etc.). A useful first step is a semanticlabelling step that is used to segment fruit from background andgenerate per pixel labels corresponding to the parts of the fruit (e.g.calyx, body, achene, etc.). In the same manner as the Computer VisionSubsystem (which makes crude 3D geometry measurements before picking) 3Dgeometry can be recovered by the QC Subsystem by adapting the parametersof a generative model to maximize the agreement between the model andthe image data. Again, a statistical prior can be used to obtain amaximum likelihood estimate of the values of the shape parameters. Auseful innovation is to use an estimate of the mass density of the fruitto determine an estimate of weight from an estimate of volume. By thismeans, we obviate the need to add the extra complexity of a mechanicalweighing device.

Most aspects of quality judgement are somewhat subjective. Whilst humanexperts can grade picked fruit reasonably consistently, it may be hardfor them to articulate exactly what factors give rise to a particularquality label. Therefore, a useful innovation is to use qualitylabelling data provided by human experts to train a machine learningsystem to assign quality labels automatically to newly picked fruit.This may be achieved by training an image classifier with training datacomprising (i) images of the picked fruit obtained by the QC hardwareand (ii) associated quality labels provided by the human expert. Avariety of models could be used to map the image data to a quality labelsuch as a simple linear classifier using hand-crafted featuresappropriate to the type of fruit in question. E.g. in the case ofstrawberries, appropriate features might be intended to captureinformation about geometric symmetry, seed density (which can indicatedryness of the fruit), ripeness, and surface folding. With enoughtraining data, it would also be possible to use a convolutional neuralnetwork to learn a mapping directly from images to quality labels.

FIG. 9 shows a sectional view of a typical embodiment of a QC imagingchamber. Cameras (and possibly other sensors, such as cameras sensitiveto specific (and possibly non-visible) portions of the EM spectrumincluding IR, (ii) cameras and illuminators that use polarised light,and (iii) sensors specific to particular chemical compounds that mightbe emitted by the fruit.) positioned around the walls of the cylindricalimaging chamber provide a view of every part of the surface of thepicked fruit. The picked fruit is gripped by a suitable end effector(the one shown here is the hook design described in Appendix A) andlowered into the chamber by the Picking Arm (of which only the head isshown). A useful innovation is to create a ‘chimney’ to reduce theamount of ambient light entering the QC rig—this is a small cylinder ontop of the imaging chamber's aperture that blocks unwanted light fromthe sides. One difficulty associated with imaging fruit inside the QCimaging chamber is that fruit debris can collect inside the chamber.Therefore a valuable innovation is to equip the chamber with a base thatcan be pulled out and wiped clean after a period of use.

7. Storage Subsystem

The purpose of the Storage Subsystem is to store picked fruit fortransport by the robot until it can be unloaded for subsequentdistribution. Because some types of fruit can be damaged by repeatedhandling, it is often desirable to package the fruit in a mannersuitable for retail immediately upon picking. For example, in the caseof fruits like strawberries or raspberries, picked fruits are typicallytransferred directly into the punnets in which they will be shipped toretailers. Typically, punnets are stored in trays, with 10 punnets pertray arranged in a 2-by-5 grid. When all the punnets in each tray arefilled, the tray is ready for removal from the robot and replacementwith an empty tray.

Since some fruit can be bruised easily by vibration caused by the motionof the robot over the ground, a useful innovation is to mount the traysvia a suspension system (active or passive) designed to minimize theiracceleration under motion of the robot over rough terrain.

Unloading full trays of picked fruit may necessitate the robottravelling to the end of the row—so it is advantageous for the robot toaccommodate more trays to amortize the time cost of travelling to andfrom the end of the row over more picked fruit. However, it is alsoadvantageous for the robot to be small so that it can manoeuvred andstored easily. Therefore a useful innovation is to equip the robot withtray-supporting shelves that extend outwards at each end but detach orrotate (up or down) out of the way to reduce the robot's length when itneeds to be manoeuvred or stored in a confined spaced.

Another useful innovation is to store trays in two vertically orientedstacks inside the body of the robot as illustrated in FIG. 3. One stackcontains trays of yet-to-be-filled punnets, the other stack containstrays of full punnets. The Picking Arm transfers picked fruit directlyinto the topmost trays. Once the punnets in a tray are filled, the stackof full trays descends by one tray depth to accommodate a new a tray, atray slides horizontally from the top of the stack of yet-to-be-filledtrays to the top of the stack of full trays, and the stack ofyet-to-be-filled trays ascends by one tray depth to bring a newyet-to-filled tray within reach of the robot arm. This design allows therobot to store multiple trays compactly within a limited footprint—whichis important in a robot that must traverse narrow rows of crops or betransported on a transport vehicle.

Refrigerating picked fruits soon after picking may dramatically increaseshelf life. One advantage of the compact arrangement of trays describedabove is that the full trays can be stored in a refrigerated enclosure.In practice however, the power requirements of a refrigeration unit onboard the robot may be greater than can be met readily by convenientportable energy sources such as rechargeable batteries. Therefore,another useful innovation is to use one of various means of remote powerdelivery to the fruit picking robot. One possibility is to useelectrified overhead wires or rails like a passenger train. Another isto use an electricity supply cable connected at one end to the robot andat the other to a fixed electrical supply point. The electricity supplycable might be stored in a coil that is wound and unwound automaticallyas the robot progresses along crop rows and such a coil might be storedon a drum that is located inside the robot or at the end of the croprow. As an alternative to delivering electrical power directly to therobot, a coolant liquid may be circulated between the robot and a staticrefrigeration unit via flexible pipes. In this case, the robot can usean internal heat exchanger to withdraw heat from the storage container.

FIG. 10 shows a diagram illustrating a space-saving scheme for storingtrays of punnets within a picking robot. Trays of full punnets arestored in one stack (shown right), trays of empty punnets are stored inanother (shown left). The Picking Arm can place picked fruit only in thetopmost trays. Once the right-hand side topmost tray is full (A), thestack of trays of full punnets descends downwards, a tray slidessideways (B), and the stack of empty trays ascends (C).

Tray removal/replacement may be achieved by a human operator or byautomatic means. A useful innovation is a means of drawing the humanoperator's attention to the need for tray replacement via thecombination of a strobe light on the robot itself and a correspondingvisual signal in the Management User Interface. A strobe light thatflashes in a particular colour or with a particular pattern of flashesmay be advantageous in allowing the operator to relate the visual signalin the UI to the specific robot that requires tray replacement or otherintervention. Another useful innovation is the idea of using a small,fast moving robot to work in concert with the larger, slower movingpicking robot. The small robot can remove trays (or full punnets)automatically from the picking robot and deliver them quickly to arefrigeration unit where they can be refrigerated for subsequentdistribution.

For some types of fruit, it is common for the customer (supermarketetc.) to define requirements on the size and quality of fruit in eachpunnet (or in each tray if punnets). Typical requirements include:

-   -   i. each full punnet has total weight within some allowable        tolerance of a nominal value;    -   ii. less than some proportion of the fruits in each punnet        differ in size by more than some threshold percentage from the        mean; and    -   iii. less than some proportion of the fruit in each punnet        exhibits unusual shape or blemish.

Depending on the contract between the grower and the customer, punnetsnot meeting these requirements (or e.g. trays containing one or morepunnets not meeting these requirements) may be rejected by the customer,reducing the grower's profit. The commercial requirements can bemodelled by a cost function that is a monotonically decreasing functionof the grower's expected profit from supplying a punnet or tray to acustomer. For example, a simple punnet cost function might depend on alinear combination of factors as follows:

Cost=w ₀ e++w ₁ ·u+w ₂ ·c+w ₃ ·d

Where e means the excess weight of strawberries in the punnet comparedto the target weight, u is an indicator variable that is 1 if the punnetis underweight or 0 otherwise, and c is a measure of the number ofstrawberries that are outside of the desired size range. Finally, d is ameasure of how long it will take to place a strawberry in a particularpunnet, which is a consequence of how far the arm will have to travel toreach the punnet. The weights w_(i) reflect the relative importance ofthese factors to profitability, e.g. w₁ reflects the cost of a traycontaining an underweight punnet being rejected, weighted by the riskthat an additional underweight punnet will cause the tray to berejected; similarly, w₃ reflects the impact on overall machineproductivity of spending more time placing strawberries in more distantpunnets.

An interesting observation is that distributing the exact same pickedfruits differently between punnets could give rise to a different totalcost according to the cost function described above. For example,because meeting the punnet weight or other packaging requirements moreprecisely means that less margin for error is required, so that agreater number of punnets can be filled with the same amount of fruit,or because placing similarly sized fruits in each punnet reduces thelikelihood that a tray will be rejected by the customer. Therefore, auseful innovation is a strategy for automating the allocation of pickedfruit into multiple punnets (or a discard container) based on size andquality measures to minimize the statistical expectation of total costaccording to the metric described earlier, i.e. to maximize expectedprofitability for the grower. Compared to human pickers, a softwaresystem can maintain a more accurate and more complete record of thecontents of many punnets simultaneously. Thus, the robot can placepicked fruit in any one of many partially filled punnets (or discardpicked fruit that is of insufficient size or quality). However, the taskis challenging because:

-   -   there may be room for only a limited number of partially filled        punnets;    -   as the punnets are filled up, the amount of space available for        additional fruit is reduced;    -   moving picked fruits from punnet to punnet is undesirable        because it is time consuming and may damage the fruit; and    -   the size and quality of yet-to-be-picked fruits is generally not        known a priori, and so it is necessary to optimize over possible        sequences of picked fruits and associated quality and size        grading.

In a simple embodiment of the above idea, each successive picked fruitmight be placed to maximize incremental cost decrease according to thecost metric described earlier. However, this greedy local optimizationapproach will not produce a globally optimal distribution of fruit. Amore sophisticated embodiment works by optimizing over the expectedfuture cost of the stream of yet-to-be-picked strawberries. Whist it maynot be possible to predict the size or quality of yet-to-be-pickedstrawberries, it is possible to model the statistical distribution overthese properties. This means that global optimization of fruit placementcan be achieved by Monte Carlo simulation or similar. For example, eachfruit can be placed to minimize total cost considering (i) the knownexisting placement of strawberries in punnets and (ii) expectation overmany samples of future streams of yet-to-be-picked strawberries. Aprobability distribution (Gaussian, histogram, etc.) describing the sizeof picked fruits and possibly other measures of quality can be updateddynamically as fruit is picked.

Note that the final term in the above cost function (w₃·d) can be usedto ensure that the robot tends to place larger strawberries in moredistant punnets. Since punnets containing larger strawberries requirefewer strawberries, this innovation minimizes the number oftime-consuming arm moves to distant punnets.

Sometimes, the Storage Subsystem cannot place picked fruit into anyavailable punnet without increasing the expected cost (i.e. reducingexpected profitability), for example because a strawberry is too largeto be placed in any available space, or because its quality or sizecannot be determined with high statistical confidence. Therefore,another useful innovation is to have the robot place such fruits into aseparate storage container for subsequent scrutiny and possiblere-packing by a human operator.

For fruits that are picked in bunches comprising several individualfruits on the same branch structure, e.g. table grapes or on-the-vinetomatoes, it may be important that none of the individual fruits isdamaged or otherwise blemished, for example because a single rottenfruit can shorten the life or spoil the appearance of the entire bunch.Therefore, a valuable innovation is a two-phase picking procedure inwhich first the entire bunch is picked and second unsuitable individualfruits are removed from it. In one embodiment, this might work asfollows:

-   1. A first robot arm picks the bunch by severing the stalk.-   2. Visual inspection of the bunch is performed to determine the    positions of any blemished individual fruits. During visual    inspection, the first robot arm continues to hold the bunch for    visual inspection.-   3. A second robot arm trims blemished fruits from the bunch. A    picking head similar to that used to pick individual fruits like    strawberries singly can also be used to trim individual fruits from    a bunch.

In another embodiment of this idea, a first robot arm might transfer thebunch to a static support for subsequent inspection and removal ofunwanted individual fruits. By this means, it is possible to use only asingle robot arm.

As well as deciding into which punnet (or other container) picked fruitshould be placed, it may also be necessary or beneficial for a robot todecide where in the target punnet to place the fruit. A key challenge isto place picked fruit so as to minimize bruising (or other kinds ofdamage) due to collisions with the walls of floor of the punnet or withother fruit already in the punnet. In the context of fruit pickingsystems that work by gripping the stalk of the fruit, another challengeis to determine the height at which the fruit should be released intothe punnet—too high and the fruit may be bruised on impact, too low andthe fruit may be squashed between the gripper and the base of thepunnet. Additionally, picked fruit doesn't necessarily hang verticallybecause the stalk is both non-straight and somewhat stiff. Therefore, auseful innovation is to measure the vertical displacement between thebase of the fruit and the point at which it is gripped or the pose ofthe picked fruit in an end effector coordinate system, so that the fruitcan be released at the optimal height. This can be achieved by using amonocular or stereo camera to determine the position of the bottom ofthe picked fruit relative to the (presumed known) position of thegripper. A related innovation is to use an image of the punnet (obtainedby the cameras in the Picking Head or otherwise) to determine theposition of other picked fruit already in the punnet. Then the positionor release height can be varied accordingly. For fruits such asstrawberries that may be usefully held by their stalks, another usefulinnovation is to orientate the gripper such that the stalk is heldhorizontally before the fruit is released into the storage container.This allows the compliance of the section of stalk between the gripperand the body of the fruit to be used to cushion the landing of the fruitwhen it is placed into the container.

A related innovation is to position and orientate the fruitautomatically to maximize visual appeal. This might be achieved, forexample, by placing fruit with consistent orientation.

Because the robot knows which fruit was placed in each punnet, it cankeep a record of the quality of the punnet. Therefore, a usefulinnovation is to label each punnet with bar code that can be read by therobot and therefore used to related specific back to the record of whichfruit the punnet contains.

8. Mapping Subsystem

A human supervisor uses the Management User Interface to indicate on amap where robots should pick (see below). A prerequisite is ageo-referenced 2D map of the environment, which defines both (i) regionsin which robots are free to choose any path (subject to the need totraverse the terrain and to avoid collision with other robots) and (ii)paths along which the robot must approximately follow, e.g. between rowsof growing plants. The robot can pick from plants that are distributedirregularly or regularly, e.g. in rows.

A suitable map may be constructed by a human supervisor using theMapping Subsystem. To facilitate map creation, the Mapping Subsystemallows a human operator easily to define piecewise-linear paths andpolygonal regions. This is achieved by any of several means:

-   -   Using geo-referenced aerial imagery and image annotation        software. The UI allows the user to annotate the aerial imagery        with the positions of the vertices of polygonal regions and        sequences of positions defining paths, e.g. via a series of        mouse clicks. When annotating the start and end points of rows        of crops, an integer-based row indexing scheme is used to        facilitate logical correspondence between the start and end        locations.    -   Using a physical survey device, the position of which can be        determined accurately, e.g. via differential GPS. The user        defines region boundaries by positioning the surveying tool        manually, e.g. at a series of points along a path, or at the        vertices of a polygonal region. A simple UI device such as a        button allows the user to initiate and terminate definition of a        region. The survey device may be used to define the physical        locations of (i) waypoints along shared paths, (ii) the vertices        of polygonal regions in which the robot can choose any        path, (iii) at the start and end of a row of crops. The survey        device may be a device designed for handheld use or a robot        vehicle capable of moving under radio remote control.

In the context of farming, an important concern is that heavy robots maydamage soft ground if too many robots take the same route over it (or ifthe same robot travels the same route too many times). Therefore, auseful innovation is to choose paths within free regions to distributeroutes over the surface of the ground as far as possible. A tuneableparameter allows for trade-off between travel time and distance and thedegree of spread.

9. Management Subsystem

The Management Subsystem (including its constituent Management UserInterface) has several important functions:

-   -   It allows a human supervisor to define which rows of crops        should be picked using a 2D map (created previously using the        Mapping Subsystem).    -   It allows a human supervisor to set the operating parameter        values to be used during picking and QC, e.g. the target ranges        of fruit size and ripeness, the quality metric to be used to        decide whether to discard or keep fruit, how to distribute        fruits between punnets, etc.    -   It facilitates the movement of robots around the farm.    -   It divides work to be done amongst one or more robots and human        operators.    -   It controls the movement of the robot along each row of        strawberries.    -   It allows robots to signal status or fault conditions to the        human supervisor.    -   It allows the human supervisor immediately to put any or all        robots into a powered down state.    -   It allows the human supervisor to monitor the position and        progress of all robots, by displaying the position of the robots        on a map.

If there are multiple robots, then they collaborate to ensure they canmove around in the same vicinity without colliding.

In case fully autonomous navigation of the robots around a site isundesirable for safety or other reasons, it may be desirable for robotsto be capable of being driven temporarily under human remote control.Suitable controls might be made available by a radio remote controlhandset or by a software user interface, e.g. displayed by a tabletcomputer.

To obviate the need for the human operator to drive several robotsseparately (e.g. from a storage container to the picking site), avaluable innovation is a means by which a chain of several robots canautomatically follow a single ‘lead’ robot driven under human control.The idea is that each robot in the chain follows its predecessor at agiven approximate distance and takes approximately the same route overthe ground.

A simple embodiment of this idea is to use removable mechanicalcouplings to couple the second robot to the first and each successiverobot to its predecessor. Optionally a device to measure the directionand magnitude of force being transmitted by a robot's coupling to itspredecessor might be used to derive a control signal for its motors. Forexample, a following robot might always apply power to its wheels ortracks in such a way as to minimize or otherwise regulate the force inthe mechanical coupling. By this means all the robots in the chain canshare responsibility for providing motive force.

More sophisticated embodiments of this idea obviate the need formechanical couplings by using a combination of sensors to allow robotsto determine estimates of both their absolute pose and their poserelative to their neighbours in the chain. Possibly a communicationnetwork (e.g. a WiFi network) might be used to allow all robots to sharetime-stamped pose estimates obtained by individual robots. An importantbenefit is the possibility of combining possibly-noisy relative andabsolute pose estimates obtained by many individual robots to obtain ajointly optimal estimate of pose for all robots. In one such embodiment,robots might be equipped with computer vision cameras designed to detectboth absolute pose in the world coordinate system and their poserelative to their neighbours. Key elements of this design are describedbelow:

-   -   The robot is designed to have visually distinctive features with        known position or pose in a standard robot coordinate frame. For        example, visually distinctive markers might be attached to each        robot in certain pre-determined locations. The markers are        typically designed for reliable automatic detection in the        camera images.    -   A camera (or cameras) is (are) attached to each robot with known        pose in a standard robot coordinate frame. By detecting the 2D        locations in its own camera image frame of visually distinctive        features belonging to a second robot, one robot can estimate its        pose relative to that of the second robot (e.g. via the Discrete        Linear Transformation). Using visually distinctive markers that        are unique to each robot (e.g. a bar code or a QR code or a        distinctive pattern of flashes made by a flashing light)        provides means by which a robot can uniquely identify the robot        that is following it or being followed by it.    -   One or more robots in the chain also maintain an estimate of        their absolute pose in a suitable world coordinate system. This        estimate may be obtained using a combination of information        sources, e.g. differential GPS or a computer-vision based        Simultaneous Localization and Mapping (SLAM) system. Absolute        position estimates from several (possibly noisy or inaccurate)        sources may be fused to give less noisy and more accurate        estimates.    -   Inter-robot communications infrastructure such as a wireless        network allows robots to communicate with each another. By this        means robots can interrogate other robots about their current        pose relative to the robot in front. Pose information is        provided along with a time stamp, e.g. so that the moving robots        can compensate for latency when fusing pose estimates.    -   In a chain of robots, the absolute and relative position        estimates obtained by all robots are fused to obtain a higher        quality estimate of the pose of all robots.    -   A PID control system is used by each robot to achieve a desired        pose relative to the trajectory of the lead robot. Typically, a        target position for the control system is obtained by finding        the point of closest approach on the lead robot's trajectory.        The orientation of the target robot when it was previously at        that point defines the target orientation for the following        robot. Target speed may be set e.g. to preserve a constant        spacing between all robots.

When picking, teams of robots may become dispersed over a large area.Because robots are visually similar, this may make it very difficult forhuman supervisors to identify individual robots. To allow the humansupervisor to relate robot positions displayed on a 2D map in thesoftware UI to robot positions in the world, a useful innovation is toequip each robot with a high visibility strobe light that gives anindication in response to a mouse click (or other suitable UI gesture)on the displayed position in the UI. Individual robots can be made moreuniquely identifiable by using strobes of different colours anddifferent temporal sequences of illumination. A related innovation is todirect the (possibly coloured) light produced by the strobe upwards ontothe roof of the polytunnel in which the crops are being grown. Thisfacilitates identification of robots hidden from view by tall crops orthe tables on which some crops (like strawberries) are grown.

Because a single human supervisor may be responsible for multiple robotsworking simultaneously, it is useful if the UI exposes controls (e.g.stop and start) for each robot in the team. However, one difficulty isto know which remote control setting is necessary to control whichrobot. In a situation where an emergency stop is needed, therefore, thesystem is typically designed so that pressing the emergency stop button(for example on the supervisor's tablet UI) will stop all robots forwhich the supervisor is responsible. This allows the supervisor todetermine which robot was which after ensuring safety.

While picking, it is possible that robots will encounter so-called faultconditions that can be only be resolved by human intervention. Forexample, a human might be required to remove and replace a full tray ofpicked fruit or to untangle a robot from an obstacle that has caused itto become mechanically stuck. This necessitates having a humansupervisor to move from robot to robot, e.g. by walking. To allow ahuman supervisor to do this efficiently, a useful innovation is to useinformation about the position of the robots and the urgency of theirfault conditions (or impending fault conditions) to plan the humansupervisor's route amongst them. Route planning algorithms can be usede.g. to minimize the time or that human supervisors spend moving betweenrobots (and therefore to minimize the number of human supervisorsrequired and their cost). Standard navigation algorithms need to beadapted to account for the fact that the human operator moves at finitespeed amongst the moving robots.

10. Robot Control Subsystem 10.1. Overview

Whilst the robot is picking, the Robot Control Subsystem makes acontinuous sequence of decisions about which action to perform next. Theset of actions available may include (i) moving the whole robot forwardsor backwards (e.g. along a row of plants), (ii) moving the Picking Armand attached camera to previously unexplored viewpoints so as tofacilitate detection of more candidate fruit, (iii) moving the PickingArm and attached camera in the local vicinity of some candidate fruit soas to refine an estimate of its pose or suitability for picking, and(iv) attempting to pick candidate fruits (at a particular hypothesizedposition/orientation). Each of these actions has some expected cost andbenefit. E.g. spending more time searching for fruit in a particularvicinity increases the chances that more fruit will be picked(potentially increasing yield) but only at the expense or more timespent (potentially decreasing productivity). The purpose of the RobotControl System is to schedule actions, ideally in such a way as tomaximize expected profitability according to a desired metric.

In a simple embodiment, the Robot Control Subsystem might move the wholerobot and the picking head and camera in an alternating sequence ofthree phases. In the first phase, the arm moves systematically, e.g. ina grid pattern, recording the image positions of detected fruits as itdoes so. In the second phase, the picking head and camera move to aposition near each prospective target fruit in turn, gathering moreimage data or other information to determine (i) whether or not to pickthe fruit and (ii) from what approach direction to pick it. During thissecond phase, the system determines how much time to spend gatheringmore information about the target fruit on the basis of a continuouslyrefined estimate of the probability of picking a suitable fruitsuccessfully (i.e. picking success probability). When the estimatedpicking success probability is greater than some threshold, then pickingshould take place. Otherwise the control system might continue to movethe arm until either the picking success probability is greater than thethreshold or some time limit has expired. In the third phase, once alldetected fruits have been picked or rejected for picking then the wholerobot might move along the row of plants by a fixed distance.

10.2 Total Position Control

During picking, the Control Subsystem uses the Total PositioningSubsystem to move the whole robot amongst the plants and within reach offruit that is suitable for picking, e.g. along a row of strawberryplants. Typically, the robot is moved in a sequence of steps, pausingafter each step to allow any fruit within reach of the robot to bepicked. It is advantageous to use as few steps as possible, e.g. becausetime is required for the robot to accelerate and decelerate during eachstep. Furthermore, because the time required to pick the within-reachfruit depends on the relative position of the robot to the fruit, it isadvantageous to position the robot so to minimize expected picking time.Thus, a valuable innovation is to choose the step sizes and directionsdynamically so as to try to maximize expected picking efficiencyaccording to a suitable model.

In a simple embodiment of this idea, a computer vision camera might beused to detect target fruit that is nearby but outside of the robot'spresent reach. Then the robot can be either (i) repositioned so as tominimize expected picking time for the detected fruit or (ii) moved agreater distance if no suitable fruit was detected in its originalvicinity. Additionally, a statistical model of likely fruit positionsmight be used to tune step size. The parameters of such a statisticalmodel might be refined dynamically during picking.

10.3. Robot Arm Path Planning

The Picking Arm and attached camera moves under the control of theControl Subsystem to locate, localize, and pick target fruits. To enablethe arm to move without colliding with itself or other obstacles, a pathplanning algorithm is used to find collision-free paths between aninitial configuration (i.e. vector of joint angles) and a newconfiguration that achieves a desired target end effector pose. Physicssimulation based on a 3D model of the geometry of the arm and the scenecan be used to test whether a candidate path will be collision free.However, because finding a collision-free path at runtime may beprohibitively time consuming, such paths may be identified in advance byphysical simulation of the motion of the robot between one or more pairsof points in the configuration space—thereby defining a graph (or ‘routemap’) in which the nodes correspond to configurations (and associatedend effector poses) and edges correspond to valid routes betweenconfigurations. A useful innovation is to choose the cost (or ‘length’)assigned to each edge of the graph to reflect a weighted sum of factorsthat reflect the overall commercial effectiveness of the picking robot.These might include (i) the approximate time required, (ii) the energycost (important in a battery powered robot), and (iii) the impact ofcomponent wear on expected time to failure of robot components (whichinfluences service intervals, downtime, etc.). Then path planning can beconducted as follows:

-   1. Search over the nodes of the graph to find a configuration C0    that can be reached without collision by a linear move from the    initial configuration Ci.-   2. Search over the nodes of the graph to find a configuration C1    from which the target pose can be reached without collision by a    simple linear move. Configurations corresponding to the target pose    can be determined by inverse kinematics, possibly using the    configuration associated with each candidate configuration as    starting point for non-linear optimization.-   3. Find the shortest path in the graph between nodes C0 and C1, e.g.    using Djikstra's algorithm or otherwise.

One limitation of this approach is that the graph can only beprecomputed for known scene geometry—and in principle the scene geometrycould change every time the robot moves, e.g. along a row of crops. Thismotivates an interesting innovation, which is to build a mapping betweenregions of space (‘voxels’) and the edges of the route map graphcorresponding to configuration space paths that would cause the robot tointersect that region during some or all of its motion. Such a mappingcan be built easily during physical simulation of robot motion for eachedge of the graph. By approximating real and possibly frequentlychanging scene geometry using voxels, those edges corresponding to pathswhich would cause the arm to collide with the scene can be quicklyeliminated from the graph at runtime. A suitable voxel-based model ofapproximate scene geometry might be obtained using prior knowledge ofthe geometry of the growing infrastructure and the pose of the robotrelative to it. Alternatively, a model may be formed dynamically by anyof a variety of means, such as depth cameras, ultrasound, or stereovision.

In the context of fruit picking, some kinds of collision may not becatastrophic, for example, collision between the slow-moving arm andperipheral foliage. Therefore, another useful innovation is to use apath planning algorithm that models obstacles not as solid object(modelled e.g. as bounding boxes), but as probabilistic models of scenespace occupancy by different types of obstacle with different materialproperties, e.g. foliage, watering pipe, grow bag. Then the pathplanning algorithm can assign different costs to different kinds ofcollision, e.g. infinite cost to a collision with an immovable objectand a lower (and perhaps velocity-dependent) cost to collisions withfoliage. By choosing the path with the lowest expected cost, the pathplanning algorithm can maximize motion efficiency, e.g. by adjusting atrade-off between economy of motion and probability collision withfoliage.

10.4. Learning Control Policies by Reinforcement Learning

Given an estimate of the approximate location of target fruit, thesystem can gain more information about the target (e.g. its shape andsize, its suitability for picking, its pose) by obtaining more viewsfrom new viewpoints. Information from multiple views can be combined tocreate a more accurate judgement with respect to the suitability of thefruit for picking and the suitability of a particular approach vector.As a simplistic example, the average colour of a target fruit inmultiple views might be used to estimate ripeness. As another example,the best viewpoint might be selected by taking the viewpointcorresponding to the maximum confidence estimation of stalk orientation.However, obtaining more views of the target may be time-consuming.Therefore, it is important (i) to choose the viewpoints that will beprovide the most useful additional information for the minimum cost, and(ii) to decide when to stop exploring more viewpoints and attempt topick the target or abandon it. For illustration:

If a target fruit (or its stem) is partially occluded (by foliage, otherfruits, etc.) then it may be valuable to move in the direction requiredto reduce the amount of occlusion. Generally, it is desirable to find aviewpoint from which the whole fruit is visible without occlusionbecause such a viewpoint defines, via the back-projected silhouette, avolume of space in which the picking head (and picked fruit) can bemoved towards the target fruit without colliding with any otherobstacles.

If a target fruit is observed from a viewpoint that makes it hard todetermine its pose (or that of its stalk) for picking purposes, then itmay be valuable to move to a viewpoint from which it would be easier todetermine its pose.

If the target fruit is positioned near other target fruits so that whichstalk belongs to which fruit is ambiguous then it may be valuable toobtain another view from a viewpoint in which the stalks can be moreeasily associated with target fruits

If approaching the fruit to pick it from the current viewpoint wouldrequire a time-consuming move of the Picking Arm (e.g. because somemoves require significant reconfiguration of the robot arm's jointangles) then it would be desirable to localize the strawberry from aviewpoint corresponding to a faster move.

Sometimes multiple views will be required to determine with sufficientconfidence that the fruit should be picked. Sometimes, it will beunambiguous but rather than obtaining more views it would be better tomove on, e.g. leaving the fruit can be picked by human pickers instead.However, designing an effective control routine manually may beprohibitively difficult.

A strategy for doing this is to use reinforcement learning to learn acontrol policy that decides on what to do next. The control policy mapssome state and the current input view to a new viewpoint. The statemight include observations obtained using previous views and theconfiguration of the arm, which could affect the cost of subsequentmoves.

To train a control policy via reinforcement learning, it is necessary todefine a utility function that rewards success (in this case pickingsaleable fruit) and penalizes cost (e.g. time spent, energy consumed,etc.). This motivates the idea that a control policy could be trainedwhilst robots operate in the field, using high quality picking successinformation using their on-board QC rig to judge picking success. Aninteresting innovation is that multiple picking robots can be used toexplore the space of available control policies in parallel, sharingresults amongst themselves (e.g. via a communication network or centralserver) so that all robots can benefit from using the best-known controlpolicy. However, one limitation of that approach is that it may take agreat deal of time to obtain enough training data for an effectivecontrol policy to be learned. This gives rise to an importantinnovation, which is to use for training purposes images of the sceneobtained from a set of viewpoints arranged on a grid in camera posespace. Such a dataset might be captured by driving the picking robotunder programmatic control to visit each grid point in turn, acquiring a(stereo) image of the scene at each one. Using a training set acquiredin this way, reinforcement learning of a control policy can be achievedby simulating the movements of the robot amongst the availableviewpoints, accounting for the costs associated with each movement. Insimulation, the robot can move between to any of the viewpoints on thegrid (under the control of current control policy), perform imageprocessing on each, and decide to pick a target fruit along ahypothesized physical path. It is not possible to be certain thatpicking a target fruit along a particular path have succeeded in thephysical world. However, for some target fruits, merely identifying thecorrect stalk in a stereo viewpoint gives a high probability of pickingsuccess. Therefore, we use correct identification of the stalk of a ripefruit as a proxy for picking success in reinforcement learning. Groundtruth stalk positions may be provided by hand for the training set.

Central to reinforcement learning is some means evaluating theeffectiveness of a particular control policy on the dataset. In outline,this is achieved as follows:

1. Set cost=02. For each target fruit in dataset3. Start at a randomly selected nearby viewpoint on grid4. Update state (includes current pose and suitability for pickingestimate) using current view5. Use current control policy to map state to action (in {Move, Pick,Abandon})

6. Switch(action)

Move:

-   -   i. Move Picking Arm to new viewpoint (determined by current        policy)    -   ii. Increase cost by cost of move (function of time taken, power        consumed, etc.)    -   iii. Goto 3

Pick:

-   -   i. Increase cost by cost of moving Picking Arm along picking        trajectory (determined by current pose estimate)    -   ii. If picking was successful, decrease cost by value of        successful pick    -   iii. Goto 1 and select next target

Abandon:

-   -   i.Goto 1 and select next target

Using this cost evaluation scheme, we can compare the effectiveness ofmultiple control policies and select the best, e.g. by exhaustive searchover available policies.

10.5. Holistic Robot Control

The reinforcement learning strategy described above relates to controlpolicies for localizing a fruit and determining its suitability forpicking given an approximate initial estimate of its position. Notehowever that the same approach can also be used to train a holisticcontrol policy for the whole robot. This means expanding space ofavailable actions to include (i) moving the Picking Arm to more distantviewpoints (to find coarse initial position estimates for target fruits)and (ii) moving by a given amount along the row of crops. An additionalinnovation is to extend the reinforcement learning scheme to includeactions carried out by human operators, such as manual picking of hardto reach fruit.

11. Miscellaneous Innovations

-   1. Because picking robots maintain a continuous estimate of their    position in a map coordinate system, they can gather geo-referenced    data about the environment. A useful innovation is therefore to have    robots log undesirable conditions that might require subsequent    human intervention along with a map coordinate and possibly a    photograph of the scene. Such conditions might include:    -   damage to the plant or growing infrastructure (e.g. caused by a        failed picking attempts or otherwise);    -   the decision to leave ripe fruit unpicked because picking would        incur too great a risk of failure or because the fruit is out of        reach of the Picking Arm.-   2. A useful and related idea is to have picking robots store the map    coordinate system locations of all detected fruit (whether ripe or    unripe) in computer memory. This makes possible several innovations:    -   a. One such innovation is to perform yield mapping to enable the        farmer to identify problems such as disease or under- or        over-watering early. Of interest might be e.g. the density of        fruit production or the proportion of unripe fruits that        subsequently mature into ripe fruits.    -   b. Another innovation is yield prediction. To pick ripe fruits        in good time, picking robots must typically traverse every crop        row every few days. By acquiring images of ripe and unripe        fruits, they can measure the size of individual target fruits as        they grow and ripen. Since most unripe fruits will ripen in        time, such data facilitates learning predictive models of future        crop yield, e.g. in the next day, week, etc. A suitable model        might map current and historic ripeness and size estimates for        individual target fruits and weather forecast information (e.g.        hours of sunlight, temperature) to a crop yield forecast.    -   c. Another related innovation reduces the amount of time that        robots must spend searching for target fruit in the target        detection phase of picking. During target detection, the system        determines the approximate position of ripe and unripe target        fruits. The system typically finds target fruits by moving a        camera on the end of the Picking Arm(s) to obtain images from a        wide range of viewpoints. However, spending more time searching        for target fruits may increase yield but only at the expense of        a reduction in picking rate. Therefore, a useful innovation is        to store the map coordinate system position of unripe fruits        that have been detected but not picked in computer memory so        that a robot can locate not-yet-picked target fruits more        quickly on a subsequent traversal of the crop row. In a simple        embodiment, the position of previously detected but not yet        picked fruits is stored so the robot can return directly to the        same position on the subsequent traversal without spending time        searching. In a more complex embodiment, previous detections are        used to form a probability density estimate that reflects the        probability of finding ripe fruit at a particular location in        map coordinate system space (e.g. by kernel density estimation        of otherwise). This density estimate might be used to help a        search algorithm prioritize regions of space where ripe fruits        are likely to be found and deprioritize regions where they are        not. E.g. a simplistic search algorithm might obtain views from        a denser sample of viewpoints in regions of space where ripe        fruits are likely to be found. In a related innovation, yield        prediction (as in (b) above) might be used to account of the        impact on time on the ripeness of previously unripe fruits.-   3. The picking robot can be equipped to perform several functions in    addition to picking, including the ability to spray weeds or pests    with suitable herbicides and pesticides, or to reposition trusses    (i.e. stalk structures) to facilitate vigorous fruit growth or    subsequent picking.-   4. Using adjustable arms that can be repositioned to maximize    picking efficiency for a particular variety of strawberry or phase    of the growing season. Picking Arms can also be positioned    asymmetrically to increase the reach of the two arms working in    concert (see FIG. 5).-   5. In use, it is likely that a robot will lean to one side or other    due to ground slope or local non-smoothness of the terrain. This    effect may be exacerbated by the use of a suspension system    (intended to reduce shock as the robot travels over bumps) because    of the compliance of the suspension. An unfortunate consequence of    leaning over is that the position of the robot's Picking Arms will    not be positioned as designed with respect to the plants (they may    be e.g. closer or further, higher or lower). This compromises    performance because (i) pre-defined camera poses chosen in the    chassis coordinate frame may no longer be optimal if the chassis is    rotated and (ii) chassis-relative models of environment geometry    (which are used to prevent collisions between Picking Arms and the    environment) may also be wrong. One obvious strategy for    compensating for the impact of lean on the environment geometry    model is to dilate the environment geometry in 3D to provide some    tolerance to error, however this reduces performance by compromising    the available working volume for the arm (in narrow crop rows, space    may already be at a premium). Therefore a valuable innovation is a    system both to measure the degree to which the robot is leaning over    and then to compensate for the degree of the lean by adapting models    of the scene's geometry and camera viewpoints accordingly. The    degree of lean might be measured directly using an accelerometer    (which measures the vertical direction) or indirectly by measuring    the position of a part of the robot (e.g. using a vector cable    follower arm or GPS) in a coordinate frame based on the crop row.    The lean may then be allowed for by applying an appropriate 3D    transformation to pre-defined camera poses and environment geometry.    Another means of correcting for lean induced by sloping but smooth    terrain is to adjust dynamically the lateral position of the robot's    tracks in the row so that the Picking Arms are closer to their    design position despite the lean.-   6. If a picked fruit is gripped by its stalk, then it may swing like    a pendulum following repositioning by a robot arm. This may incur a    productivity cost, since for some operations (e.g. placing the fruit    into a QC imaging chamber or punnet) the pose of the fruit must be    carefully controlled to avoid collisions—which may necessitate    pausing arm motion following repositioning until the amplitude of    such oscillations decreases to an acceptable level. Therefore a    valuable innovation is to use a means of damping to reduce the    amplitude the oscillations as soon as possible. In one embodiment,    damping is achieved passively, using a soft gripper. In another    embodiment, damping is achieved actively by modulating the velocity    of the robot arm's end-effector in such a way as to bring the    oscillation to a stop as quickly as possible. An estimate of the    mass and pendulum length of the strawberry can be used to design a    deceleration profile (dynamically or otherwise) required to minimize    the amplitude or duration of oscillation.-   7. Many of the diseases and other defects that affect soft fruits    (and reduce their quality grade) affect their visual appearance in    the images obtained by our QC imaging chamber. The first image    processing step is to segment in each view the body of the fruit    from the calyx (and the background of the imaging chamber).    Typically this is achieved by using a decision forest to label    pixels. It may also be useful to segment achenes (seeds) at the same    time as described below. Then we characterise the appearance of the    body of the fruit using a variety of quality measures (which may be    innovative in isolation or combination). Some features not    previously mentioned include:    -   a. The spatial distribution of achenes (e.g. in strawberries) or        drupelets (e.g. in raspberries). For healthy fruits, achenes and        drupelets are generally arranged quite regularly, i.e. the        distances between neighbouring achenes and drupelets are        generally similar locally. However, some diseases and other        problems have the effect of disrupting this regular arrangement        in the developing fruit. Therefore an innovative idea is to use        a measure of the regularity of the spatial arrangement of        achenes or drupelets as an indicator of the quality of the        fruit. One approach is to first detect (in images obtained by        our QC imaging chamber) the image positions of achenes or        drupelets using computer vision and then assign a cost at each        such point using an energy function that assigns lowest energy        to regularly arranged points. Achenes can be detected by        semantic labelling, e.g. by using a decision forest classifier        to assign a table to each pixel. Drupelets may be detected e.g.        by using a point light source to induce a specular reflection on        the shiny surface of the fruit and then detecting local maxima        in image brightness. Then the sum of costs over points (or local        clusters of points) can be used to give an indication of fruit        health.    -   b. The colour of the achenes. In e.g. strawberries the achenes        become redder when the fruit becomes overripe and black when the        fruit becomes rotten. Labelled training images of fruits        exhibiting particular defects can be used to determine        thresholds of acceptability for colour.    -   c. The colour of the flesh of the fruit (excluding the achenes).        This is a good indicator of under-ripeness, over-ripeness, and        localised bruising. Again labelled training data can be used to        determine thresholds of acceptability.    -   d. The 3D shape of the fruit. An explicit 3D model of the fruit        may be obtained by model fitting to the silhouette and image        intensity information as described in the existing provisional        application. However an ‘implicit’ 3D model of the shape may be        obtained more simply by characterising the 2D shape in each of        multiple views. An effective strategy for doing this proceeds        one 2D view at a time by first determining the approximate long        axis of the fruit and then by characterising shape in a        coordinate system aligned with the long axis. The long axis        position may be estimated e.g. by the line joining the centroid        of the body of the fruit and the image position of the gripper        that is used to hold the fruit. Shape may be characterised e.g.        by measuring the distance from the centroid of the fruit to the        edge of the fruit in each of the clock face directions between 2        o'clock and 10 o'clock given a 9-element vector (noting that we        ignore the top of the clock face to avoid the calyx). We can        distinguish good (grade 1) shapes from bad (grade 2) shapes        using a body of training examples with associated expert-derived        ground truth labels. A suitable strategy is k nearest neighbours        in the 9D shape space. Robustness to errors in long axis        localisation can be achieved by generating multiple 9D shape        vectors for randomly perturbed versions of the detected long        axis in the training images.-   8. Typically, a picked fruit is considered grade 2 (sub-par) or    grade 3 (reject) if any of our several independent quality measures    give a grade 2 or grade 3 score (although other means of combining    defect scores are possible). Note that an important advantage of    using image features specifically designed to detect specific kinds    of defect instead of a more black-box machine learning approach is    that doing so allows us to give the grower a clear and intuitive    explanation (or, better, visual indication) for why a particular    picked fruit received a particular quality grade. This allows the    grower to adjust meaningful thresholds of acceptability for    different kinds of defect. In practice, it is commercially very    important for growers to make different decisions about the    acceptability of different kinds of defect at different times of the    season, considering the requirements of different customers, and as    a function of productivity and demand.-   9. Another application for agricultural robots is the targeted    application of chemicals such as herbicides and pesticides. By using    computer vision to locate specific parts of the plant or instances    of specific kinds of pathogen (insects, dry rot, wet rot, etc.),    robots can apply such chemicals only where they are needed, which    may be advantageous (in terms of cost, pollution, etc.) compared to    treatment systems that require spraying the whole crop. To    facilitate doing this kind of work using a robot that is otherwise    used for picking it would obviously be advantageous for the robot to    support interchangeable end effectors, including one that can be    used for picking and one that can be used for spraying. The latter    would usually require routing pipes along the length of the robot    arm, which may be difficult. However, a useful innovation is to have    the spraying end effector contain a small reservoir of liquid    chemical—thereby obviating the need for routing pipes. This might be    achieved using a cartridge system where the robot arm would visit a    station on the chassis to gather a cartridge of chemicals (which    might be similar to the cartridges used in inkjet printers).    Alternatively the arm might visit a cartridge in the chassis to suck    the required liquid chemicals from a cartridge into its reservoir or    to expel unused chemical from its reservoir back into a cartridge.    It might be that several different types of chemical are combined in    dynamically programmable combination to achieve more optimal local    treatment, or that multiple cartridges contain multiple different    chemical combinations.

Appendix A: Produce-Picking End Effectors

This appendix describes several innovative Picking Head designs.

Background

Picking fruit necessitates having an end effector (such as may beaffixed to a robot arm) that is sufficiently small, strong, selective(so as to pick only fruit that is suitable for sale), exclusive (so asnot to pick other fruit). The picking operation must not damage plantsor growing infrastructure, e.g. grow bags. Additional constraints (lowpower, low cost, lightweight, durable) further constrain the design.

This invention solves these problems with a lightweight end effectorcapable of reliable picking. lie compact nature of the designsconsiderably facilitates separating the desired items of fruit fromunwanted items.

In general picking of produce comprises selection of the fruit to bepicked, exclusion of items not intended to be picked (including unripefruits and growing infrastructure), gripping of the fruit or its stemand separating the fruit from its parent plant. Here we disclose anumber of embodiments of the invention to mechanically select, grasp,and cut produce from the host plant.

Hook Embodiment

The first embodiment of the invention selects and excludes with a hookthat has a dynamically programmable trajectory. Under the control of thePicking Arm (or otherwise) the hook is mechanically swept through a (ingeneral dynamically) chosen volume of space, and any stems within thisswept volume are gathered into the hook. With the stem thus captured,the hook may be used to pull the target fruit away from the plant (andpotential sources of occlusion like leaves or other fruits) so thatmeasurements of picking suitability may be made (including visual,olfactory and tactile measurements). Such measurements inform a decisionto pick or release the target fruit.

The hook tight be made long and narrow (e.g. in the shape of the letterJ) so as to minimize the volute that needs to be swept out as it ismoved (e.g. along its own long axis) towards the target fruit, therebyminimizing the size of the gap needed (e.g. between leaves, stems, orother fruit) for the hook to reach the target fruit without collision.

Note that it may be advantageous to position the long axes of the hooknearly coincident with the optical axis of the Picking Head camera (ornearly in the middle of the optical axes of the two eyes of a stereocamera). Doing so simplifies the problem of finding a non-colliding pathto target fruit because any target that appears from the camera'sviewpoint to be unoccluded (by other fruit or foliage) can usually besafely approached by moving along the line corresponding to the raybetween the camera's optical centre and the target. This reasoningobviates the need to obtain a 3D model of the environment to plan saferoutes towards target fruit.

At this stage the hook motion may be reversed to release the producewithout damage, or the gripper and cutter mechanism may be actuated tohold the produce and separate it from the stem prior to transport andrelease.

More detailed information about the mechanical aspect of the inventionis now presented. Referring to FIG. 11, the main features of theinvention as assembled are shown. The main components of the endeffector are specifically: (1) a hook, (2) a gripper that fits withinthe hook, (3) a lower support and (4) a blade. The hook is square incross section and in conjunction with the blade forms a scissor cuttingaction. Screws and other support structures not shown.

FIG. 12 shows the hook extended relative to the grip/cut mechanism(prior to selecting or picking-blade omitted). FIG. 13 shows the hookretracted relative to the gripper/cutter. the gripper fitting againstthe hook to grip the plant stem and the cutting mechanism in its postactuation configuration (blade omitted). FIG. 14 shows an exploded viewof the main parts in this embodiment, including the individual parts ofthe end effector including (4) the blade above the hook.

Referring to FIGS. 12-14, the main features are: (item 1) the hook(comprising a long thin section and a hook, which is semi-circular inthis embodiment), which (through movement in space, FIG. 15 item 5)allows produce selection and exclusion of unwanted items. The inside ofthe hook is shaped to form one half of the gripper mechanism, and formsone half of the scissor style cutting surface. The tip is pointed inthis embodiment, maximising selectivity and exclusivity and assistinglocation within the gripper/blade mechanism when retracted.

There is a lower support (FIGS. 11-13 item 3) which constrains theretracted hook, strengthening the gripping action and constraining theblade to be adjacent to the hook cutting surface, increasing cuttingreliability. In this embodiment the gripper (2) is shaped to fit intothe inside surface of the hook, allowing the mechanism to be maximallycompact. This embodiment uses a flexure spring.

FIG. 15 is an example of the invention showing the movement of the hook(5) to effect a capture of the plant stem. The movement of the hook (5)captures the stem within the hook. Instead of having a cutting/grippingmechanism of fixed size, in the invention the hook is small (allowinggood selection with maximal exclusion of unwanted stems) while the hookmovement is of variable size (allowing selectivity in the presence ofstem positional uncertinty).

Varying the capture volume (per item of produce) allows an optimumtrade-off between selectivity (of the desired produce) and exclusivity(of unwanted items).

The grip and cut actions are performed in the same movement by actuationof the device (refer to FIGS. 12 and 13). As the hook retracts relativeto the gripper, the stem is gripped first between the hook and thegripper (FIGS. 11-13, items 1 and 2). The gripper includes a spring,allowing for a range of stem size and allowing the hook to continue toretract. After the grip has been achieved, the continued movement of thehook against the blade separates the produce from the parent plant.Release of the fruit is achieved by extending the hook once more.

Construction method: Referring to FIG. 11. in the present embodiment,the hook (1) is made of metal, e.g. steel. The gripper (2) is made ofplastic (perhaps acetal if integrated flexure is required). The blade(4) is made of knife steel and the lower support (3) is plastic.

FIG. 16 shows the plant stem thus captured. FIG. 17 shows the producegripped and cut from the parent plant (optional operation). FIG. 18shows the release operation (optional operation).

FIG. 19 shows the sequence of operations that constitute the pickingprocess. These are as follows:

-   -   Detect Produce is performed by the Control and Computer Vision        subsystems described in the main body of this document.    -   Approach of the plant stem is made, locating the hook close to        the plant stem of the produce desired to be picked.    -   Select Item is performed using a looping movement that is        determined according to the location of the desired produce and        unwanted produce and infrastructure. An example of this movement        is shown in FIG. 15 item 5.    -   Decision Step 1 is performed with the selected item captured in        the hook (as shown in FIG. 16). With the fruit thus isolated        (but still attached to the parent plant), a check (visual,        olfactory, tactile) is made to detect the nature (and        correctness) of the fruit. The result of this step is a decision        to pick (produce for harvest or disposal) or don't-pick (unripe        fruit, captured infrastructure, nothing captured, incorrect        capture). Initial grading and quality control of the item is        performed and stored and used if Decision Step 2 is reached.    -   Release is performed in the case of undesired items being        captured by the hook, with a motion that is the reverse of        capture (FIG. 18).    -   Pick is performed in the case of successful capture of produce        ready to harvest. The hook is retracted relative to the        gripper/cutter (transitioning from configuration as in FIG. 12        to configuration as in FIG. 13). This effects a grip operation        followed by a cut operation, separating the fruit from the        parent plant (FIGS. 16 and 17 illustrate before and after        state).    -   Decision Step 2 is a secondary sensing operation performed with        the fruit picked and transferred to another part of the machine.        The result of this decision is to store or dispose of the fruit.        This decision is performed with different sensors to that in        Decision Step 1. This forms a more detailed assessment of the        fruit thus picked. The result of this step (in conjunction with        information from Decision Step 1) is a decision to store or        dispose of the fruit. In the case of store, size and quality        grading is used to determine the location of storage.    -   Store the fruit is moved to the storage area (according to        size/quality grading from Decision Steps 1 and 2) and released        with the reversal of the picking operation (extending the hook        relative to the gripper, releasing the fruit).    -   Dispose is performed in the case of picked fruit that is mouldy        or otherwise unsuitable for sale. The release operation is as        per Store above, with the exception that the transfer is to a        disposal area.

Loop Embodiment

In a second embodiment of the invention the select and exclude phases ofthe picking sequence described earlier is performed by means ofactuating a loop e.g. of wire. The diameter, position and orientation ofthe loop is programmatically controlled and actuated such that it may:

-   -   i. have an arbitrarily small volume on its approach to the        target fruit;    -   ii. increase in diameter, to be larger than the estimated        diameter of the target fruit, as it is moved in parallel with,        and centred on, the major axis of the target fruit and in the        direction of the juncture between the target fruit and its stalk        and;    -   iii. have an arbitrarily small diameter once it has moved past        the juncture of the stalk and target fruit.

In this way the loop will select the stalk of the target fruit fromother objects in the environment (e.g. other produce, leaves, growinginfrastructure etc.) the stalk can then be manipulated such that thetarget fruit may be moved away from other objects in the environment.

The fruit, if rejected after Decision Step 1, may be released by:

-   -   i. increasing the diameter of the loop to at least the estimated        diameter of the target produce; and    -   ii. moving the loop in parallel, and centred on, the major axis        of the target produce and away from the juncture of the stalk        and target produce.

Once the produce is released a new picking operation may be started.

Jaw Embodiment

In a third embodiment of the invention, the select, exclude, and gripphases of the picking sequence described earlier are performed by a setof jaws. The position, orientation and attitude (by which is meantwhether opened, partially closed or closed) of the jaws areprogrammatically controlled and actuated such that

-   -   i. they approach the stalk of the target fruit in the closed        attitude so as to minimize their swept volume;    -   ii. when arbitrarily near the stalk they are actuated to an open        attitude;    -   iii. when adjacent to the stalk they are in a partially closed        attitude so as to perform the select, exclude and grip phases        simultaneously; the jaws may be further actuated to a closed        attitude, such that a mounted blade is moved perpendicular to        the stalk, cutting it using a scissor-like motion against the        opposing side of the jaw.

The fruit, if rejected after Decision Step 1, may be released in thethird embodiment of the invention by, following step (iii) in the aboveparagraph:

-   -   i. actuating the jaw to an open attitude; and    -   ii. moving the jaw along the previous approach vector and in the        opposite direction. The produce is then released and the picking        operation may be restarted.

Loop-and-Jaw Embodiment

As shown in FIGS. 24-27, in a fourth embodiment of the invention, thesecond and third embodiments are combined such that a loop performs theselect and exclude steps and the jaws perform the grip and the cutsteps.

FIG. 20 shows the main mechanical constituents of the loop and jawassembly (the actuation mechanism of the jaw is omitted for clarity).

FIG. 21 shows the loop and jaw assembly, shown with component 11, theloop, extended.

FIG. 22 shows an exploded diagram of the main constituent parts of theloop and jaw assembly (the loop is omitted for clarity).

FIG. 23 shows the components of the loop actuation mechanism. The loopis extended and retracted by means of the rotation of the drum (16).Both the drum and loop (11) sit within housing (12) and (17), toconstrain the motion of the loop during actuation.

FIG. 24 shows the loop/jaw assembly on its approach vector towards thetarget fruit.

FIG. 25 shows the loop extended and the assembly moving in parallel tothe major axis of the target produce and in the direction of thejuncture between stalk and fruit.

FIG. 26 shows the loop having travelled past the juncture of the stalkand the fruit, the target produce is now selected and decision step 1(FIG. 19) may be applied.

FIG. 27 shows the loop is retracted to control the position of thetarget produce and the jaw is actuated so as to grab and cut the stalkof the fruit in a scissor-like motion.

More detailed information about the mechanical aspect of theloop-and-jaw embodiment of the invention are now presented. FIG. 20shows the main mechanical components of the loop and jaw assembly,being: (11) a loop which is sufficiently flexible in its plane ofoperation—being the plane perpendicular to the major axis of the targetproduce; (12) a housing for the loop to constrain the degrees of freedomof the loop to its plane of operation; (13) a blade mounted staticallyw.r.t one side of the jaw; (14) one side of the jaw, which may be sprungso as to allow stalks of various diameters to be gripped; (15) the otherside of the jaw, which forms the opposing side of grabbing mechanism andthe scissor-like cutting mechanism; (16) a drum, to which the loop ismounted, that by its rotation extends or retracts the loop; and (17) ahousing for the drum which constrains the loop to move within thegrooves of the drum.

These components (except 11 and 12) are displayed with greater clarityin the exploded diagram, FIG. 22.

The loop itself (11) may be embodied in several different ways. Incommon they have the properties that they are flexible in the plane ofoperation, and are resistant to plastic deformation. This may be in theform of a single strand or multi-strand metal or plastic wire, a seriesof static links joined such that adjoining links pivot about a commonaxis, or a series of static links that overlap adjoining links and pivotabout a flexible, continuous member that is attached to each link.

The housing for the loop (12) is shown as a hollow tube. It is mountedstatically w.r.t the superstructure of the loop assembly. It has theproperties that its inner surface has a low coefficient of friction andthat it has a cross sectional shape that matches that of the loop. Thehousing may be made of plastic, rubber, or metal and may be reinforcedwith strands of metal wire and may be lined with another material. Itmay also be in the form of a grove cut within the superstructure of theloop assembly and may be of arbitrary cross-sectional shape.

Components 16 and 17 constitute one embodiment of the actuation of theloop. It is comprised of a drum (16) which rotates within a housing(17). FIG. 18 shows detail of the drum and its assembly within thehousing. The drum has two grooves in its outer circumference and thehousing has matching grooves. When assembled these groves constrain themovement of the loop to be static with respect to the drum and slidealong the inner circumference of the housing. In this embodiment thehousing is shown mounted perpendicular to the plane of operation of theloop. However, it may also be mounted in the plane of operation of theloop so as to accommodate embodiments of the loop that are inflexible inother planes (e.g. a series of pivoted links). The actuation may haveother embodiments for example a linear actuation that pulls the loop toretract it and which is opposed by a sprung arm to extend it.

The jaws of the jaw assembly (14 and 15), may be constructed of acetal,for integrated flexure, or of less flexible material, either plastic ormetal, and include a sprung and pivoted subcomponent or have arubberised inner surface to provide compliance. This compliance is anecessary property of the jaw assembly to accommodate the gripping ofstalks of varying diameter with a gripping force of similar magnitude.The jaws are mounted onto a superstructure about which they may pivot in1 degree of freedom in the same plane. They may be actuated in a varietyof ways, one embodiment of which is a Bowden cable, another is by meansof a servo motor driven toothed-gear. A blade (13) is mounted rigidly tothe compliant jaw (14) but this compliance is not static w.r.t theblade. When the jaws are moved to the closed attitude the blade slidesover the surface of the opposing jaw (15) creating a scissor-likecutting mechanism between them.

Miscellaneous Innovations

Variations of the hook movement (FIG. 15 item 5) may be used. E.g.rotation of the hook along its major axis by approximately 90 degreesallows the hook itself to be approximately parallel to the produce stem,allowing for greater selectivity amongst stems that are close to eachother. This rotation is reversed before the gripping and cuttingactuation.

To increase the speed of picking, an end effector holding picked fruit(or a part of the end effector that is responsible for holding thefruit) may be detached and transferred to another part of the machine,and another copy (or variation) of the end effector may be used to pickmore produce. In this way, picking can work in parallel with the storageor disposal operations, thus increasing speed. Additionally, one of avariety of end effectors may be selected according to which is bestsuited to the task of picking a particular item of fruit to be picked.

The hook and gripper may capture the produce into a smaller grippedpallet that may be removed from the end effector. This comprises thegripper part (FIG. 11 Item 2) along with the hook, a subset of the hook,or an additional part (that may fit inside the hook as presented) thatmay be released from the machine.

The hook may be reconfigured to be a dual bifurcating hook, withgripper/cutter on each side. This allows the capturing hook movement tobe clockwise or counter clockwise (a variation on FIG. 15 item 5).

The blade may be in two places: above and below the gripper (separatelyactuated). By twisting the hook through approximately 180 degrees beforeoperation and using the appropriate blade, the hook movement may againbe clockwise or counter clockwise while still allowing the blade to beabove the gripper, effecting correct holding of the produce by the stem.

After picking, fruit may be lowered into an imaging chamber for gradingpurposes (see the Quality Control section, above). It is usuallydesirable that the Picking Head can be tilted downwards whilst holdingthe fruit in the imaging chamber (as illustrated in FIG. 9) becauseotherwise the design of the imaging chamber would be compromised by theneed to avoid mechanical interference with the Picking Head. Therefore,for end effectors that hold picked fruit by its stalk, a usefulinnovation is to have the surfaces that contact the stalk oriented at anangle to the vertical direction when the Picking Head is oriented suchthat there is no downwards tilt. This allows the picked fruit to hangvertically when the picking head is pitched downwards compared to thehorizontal.

The hook may be reshaped to be square, triangular, or other shapes.Instead of a hook, a simple ‘L’ shape suffices and allows for easierrelease of the fruit or other items during the (optional) release stage(FIG. 19, “Release”).

The cutter may be replaced with a less sharp blade, for a morescissor-like rather than cutting action.

The gripper may be made of rubber (allowing use without a spring) orother materials, and positioned above or below the hook, although thisis less optimal for compactness.

The cross section of the hook may be varied. Generally, a flat insidesurface is preferred to ensure a reliable cutting action.

A related innovation is a hooking apparatus designed to allow theproduce to be rotated for more complete inspection at the point ofdeciding whether to pick (FIG. 19). For example by the mechanism oftwisting its stem to allow the reverse of the produce to be imaged priorto committing to picking.

Should the Picking Arm collide with immovable objects (e.g.infrastructure) during picking, it may be able to stop automatically,for example by detecting that its intended position is different to itsactual position. However, the collision may still cause damage to thearm or its end effector, or necessitate a time-consuming intervention bya human supervisor to move the arm safely away from tangled obstacles.The probability of collision can be considerably reduced by designingthe end effector and its motion path to minimize the volume of 3D spaceswept out whilst moving towards the target. However, it is stillpossible that the front of the end effector may collide with animmovable obstacle. In this event, the following innovationssignificantly help to reduce the likelihood of damage to the robot or toinfrastructure and the requirement for human intervention:

-   -   The end effector may be designed to deform under compressive        force. The Loop design (above) self-evidently embodies this idea        if the wire loop is sufficiently deformable. If the Hook design        is used, the hook can be designed to buckle elastically or        plastically if a sufficient compressive force is applied in the        longitudinal direction.    -   By approaching target fruit by moving the end effector        approximately in the direction of the normal to the plane in        which its cross-sectional area is minimized, the probability        that any obstacles will collide first with the deformable hook        instead of other non-deformable parts of the robot is increased.    -   As an alternative to making the end effector deformable, a rigid        end effector can be mounted to a spring so that it will move        backwards into the picking head if enough force is applied.        Furthermore, a microswitch may be used to detect backwards        movement of the end effector so that the arm can stop moving (or        reverse its direction of motion) as soon as a collision occurs.        The length and stiffness of the spring should be calibrated so        that the robot arm can stop harmlessly (and subsequently        reverse) before excessive force is applied to the obstacle.    -   For the Hook embodiment, another useful innovation is to retract        the hook into its support when not actively picking to reduce        the chances of snagging.

Appendix B: A Rotary Cable Management System for a Robot Arm

In what follows, we describe an innovative solution to the problem ofrunning various types of cable through the articulating joints of robotarms. Here, cable should be interpreted to mean any flexible objectintended to guide matter or energy along its path, either as a means ofproviding power or transmitting information or moving material. Thisdefinition obviously includes (but is not limited to) electrical cablesand wires, optical fibres, and pipes.

An important challenge here is to allow a wide enough range of angularmotion at each joint. This is especially important in robots that do notmerely repeat pre-programmed motion paths but instead determinedynamically where to move as a function of observations of theenvironment. In the former case, it is usually possible to design themotion paths so that the joints are never driven past their limits. Butin the latter, the desired motion path might not be predictable inadvance. Sometimes it may not be possible to move the robot arm directlyfrom its current configuration to a desired target pose because therange of motion at one or more joints is insufficient. When thishappens, it may be necessary to make a more complex ‘reconfiguration’move so that the joint in question is oriented further away from its endstop. However, reconfiguration moves may be expensive in power or time,because they may require the robot to make large moves at all joints.Increasing the range of motion at the joints, decreases the probabilitythat a reconfiguration move will be required for the robot to reach anew target pose.

Design Requirements

Key design requirements for the rotary cable management system:

-   -   Allows the joint to achieve large changes of rotation angle.    -   Has high reliability (equates to low stress in the cable        components and low reversal of stress—which accelerates fatigue        and plastic yield).    -   Capable of carrying complex cables—self-guided robots sense        their environment and act based on this, meaning high data rates        are required to transfer enough information for effective        action, fast enough for efficient use of time. Typically, high        data rates require either twisted pair cables or fibre optic,        both of which are particularly sensitive to twisting and coiling        actions.    -   Compact—self-guided robotic systems that interact with their        environment need a small footprint to nimbly negotiate around        the environment.    -   Protected from the environment

Alternative solutions to these requirements are sub-optimal and include:

-   -   WiFi (or other wireless communications channel). However,        limited network bandwidth may make this difficult in        environments where multiple robots are working nearby and it is        not a solution for transmitting power or conducting matter.    -   Externally guided cables—this requires an ‘umbilical’ that can        get caught and damaged by the environment and severely limits        the ability of the robot end effector to move with respect to        the umbilical.    -   Optical data transmission and inductive power transfer—high        cost.    -   Slip rings (very high cost and size for reliable versions).

Description

FIG. 28 shows the different elements of the cable management system. Thesystem consists of a cable enclosure (a) and a central cable guide (b)that twists relative to the enclosure (c), allowing a coil of cable toexpand and contract much like a clock spring as the cable guide twists.The system may also be configured to support one or more coils of cable(d).

FIG. 29 shows drawings of the cable management system in situ within oneof the joints of an arm. The one or more cables run through articulatingjoints of the robot arm. The cable guide is designed such that the cableis ducted away through the centre to the next stage of the arm. Thecable is well supported by the cable guide, so only the well-definedcoil of cable moves. This is illustrated in FIG. 30 with a sequence ofdrawings showing the cable guide rotating within the cable enclosure.This arrangement minimises cyclical stresses on the cable as it neverreverse-bends, but merely bends slightly more or less, to accommodatethe twist.

At one end, the twisting motion is limited by the coil pulling tight. Atthe other end, the cable unwinds itself to the point where theinner-most section of cable starts to rub on the inside of the next coiland bend backwards (a form of capstan lock), rather than the whole coilcontinuing to unwind. The design must be arranged such that there issufficient margin for error both in manufacture, assembly and operation,such that these limits are never reached, to prolong the life of thecable.

FIG. 31 shows a cutaway view of cable winding. When wound at one end ofstroke the cable is pulled tight and there are more windings; and at theother the cable is pushed out to the edge of the enclosure and there arefewer windings. At both extremes (and between them) the change ofcurvature of the cable remains low, so the strain rate and fatigue seenby e.g. the copper and plastic components of the cable is low—giving along life for large overall displacements.

Specific Innovations:

-   -   Use of the arrangement of a central cable guide and enclosure to        define a coil of cable that can accommodate relative twisting of        one to the other with low fatigue of the cable.    -   Use of this arrangement with wires, electrical cable, optical        fibres, fibre optic cables, pipes, ribbon cables, individual        cores.    -   Use of this arrangement especially with “flat” twisted pair        cable, which is ideally sized to be flexible in one direction        but self-supporting in the other direction, so forming a stable        coil    -   Multiple stacks of coils can be arranged on top of each other        enabling many cables to be managed in a similar footprint, with        the cable guide ducting all cables away through the central        axis.    -   The enclosure can be made to open and close easily to facilitate        build of the cables. Among the many ways of doing this is to        include a hinge in the enclosure, and make the enclosure out of        many parts and build them around the coil.    -   Shelves can be added between the cables to provide a smooth        running-surface.    -   Can lubricate the cables to minimise wear and friction; with        added lubricant, or using inherently lubricious materials in the        cable, or an added membrane coiled with the cable to lubricate        the surfaces.    -   Can also add an extra element to the central cable guide or        inner-most section of cable to reduce the angle of contact with        the next coil along and prevent capstan lock, thus increasing        the stroke and reducing stress.    -   Screened cables are often used in data lines to improve        resistance to emission of noise; however screened cables are        often less flexible and available in fewer configurations than        unscreened. It is possible to benefit from the flexibility and        wide variety of unscreened cables, while maintaining high        integrity, by modifying this assembly to perform the screening        function.    -   Screening the assembly involves making the key components        (enclosure, cable guide, shelves) from metal,        conductively-filled material (e.g. carbon-loaded plastic) or        conductively-coated material (e.g. metallised plastic), and        earthing them; or by placing the whole assembly within a        conductive shell. Effective screening is likely to require        continuity of an earth between rotating halves of the assembly        which can be implemented in various ways including an earthing        conductor in the coiled cable; by adding conductive compliant        surfaces to either the inner cable guide or enclosure, (e.g.        brushes or conductive compliant seal elements); and capacitive        coupling, by reducing the gaps between components to a minimum.    -   At low temperatures, cables often suffer degradation faster due        to materials nearing their glass transition temperatures. Low        temperature operation of this cable management system can be        achieved by using the cables themselves as heaters, running        current through them to keep them warm.    -   Can pump hot and cold fluid through pipes to control        temperature.    -   Can make some or all components transparent for ease of        fabrication, inspection, and maintenance.

Appendix C: Features Summary

This section summarises the most important high-level features; animplementation of the invention may include one or more of thesehigh-level features, or any combination of any of these. Note that eachfeature is therefore potentially a stand-alone invention and may becombined with any one or more other feature or features; the actualinvention defined in this particular specification is however defined bythe appended claims.

The high level features are organized into the following categories:

-   -   Robot hardware features or core robot features    -   Operational optimization features    -   End effector features    -   Computer Vision features    -   AI/machine learning features    -   Picking process features    -   Methods or applications

There is inevitably a degree of overlap between these features. Thisapproach to organising the features is therefore not meant to be a rigiddemarcation, but merely a general high level guide.

Robot Hardware Features or Core Robot Features

In this section, we summarise features which are robot hardware featuresor core robot features. The primary feature is a robotic fruit pickingsystem comprising an autonomous robot that includes the followingsubsystems:

a positioning subsystem operable to enable autonomous positioning of therobot using a computer implemented guidance system, such as a computervision guidance system;

at least one picking arm;

at least one picking head, or other type of end effector, or other typeof end effector, mounted on each picking arm to either cut a stem orbranch for a specific fruit or bunch of fruits or pluck that fruit orbunch, and then transfer the fruit or bunch;

a computer vision subsystem to analyse images of the fruit to be pickedor stored;

a control subsystem that is programmed with or learns pickingstrategies;

a quality control (QC) Subsystem to monitor the quality of fruit thathas been picked or could be picked and grade that fruit according tosize and/or quality; and

a storage subsystem for receiving picked fruit and storing that fruit incontainers for storage or transportation, or in punnets for retail.

Whilst the primary application for this system is in pickingstrawberries, raspberries and tomatoes, this approach may be re-purposedoutside of the fruit picking context. For example, it may be used forlitter picking or collecting other kinds of items. One can thereforegeneralise the system as follows:

A robotic picking system comprising an autonomous robot that includesthe following subsystems:

a positioning subsystem operable to enable autonomous positioning of therobot using a computer implemented guidance system, such as a computervision guidance system;

at least one picking arm;

at least one end effector, or other type of end effector, mounted oneach picking arm to pick or collect an item, and then transfer thatitem;

a computer vision subsystem to analyse images of the item to be pickedor stored;

a control subsystem that is programmed with or learns picking orcollecting strategies;

a quality control (QC) subsystem to monitor the item that has beenpicked/collected or could be picked/collected; and

a storage subsystem for receiving picked/collected items and storingthat item in containers for storage or transportation.

There are multiple, optional features that can be used in such a system,or that could constitute a stand-alone feature, that can be usedindependently of the system defined above. We list these as follows.Whilst we specifically reference a fruit picking system, all of thefollowing features can be used outside of that context, for example forlitter collecting or indeed collecting other items; generalizing beyondfruit is explicitly envisaged in all that follows, throughout thisAppendix C.

A robotic fruit picking system that comprises a tracked or wheeled roveror vehicle capable of navigating autonomously using a computervision-based guidance system.

A robotic fruit picking system in which the computer vision subsystemcomprises at least one 3D stereo camera.

A robotic fruit picking system in which the computer vision subsystemthat analyses fruit images comprises image processing software fordetecting a fruit, and the control subsystem comprises software fordeciding whether to pick the fruit and the optimal strategy for pickingthe fruit, based on automatically updateable strategies, such asreinforcement learning based strategies, including deep reinforcementlearning.

A robotic fruit picking system in which the control subsystemautomatically learns fruit picking strategies using reinforcementlearning.

A robotic fruit picking system in which the picking arm has 6degrees-of-freedom.

A robotic fruit picking system in which the picking arm positions theend effector and a camera, each mounted on the picking arm.

A robotic fruit picking system in which the end effector comprises ameans of (i) cutting the fruit stalk or stem and (ii) gripping the cutstalk or stem to transfer the fruit to the QC and storage subsystems.

A robotic fruit picking system in which the robot automatically loadsand unloads itself onto, and off of, a storage container or a transportvehicle.

A robotic fruit picking system in which the robot automaticallynavigates amongst fruit producing plants, such as along rows of appletrees or strawberry plants, including table grown strawberry plants, orraspberry plants.

A robotic fruit picking system in which the system automaticallycollaborates with other robotic systems and human pickers to dividepicking work efficiently.

A robotic fruit picking system in which the system automaticallydetermines the position, orientation, and shape of a target fruit.

A robotic fruit picking system in which the system automaticallydetermines whether a fruit is suitable for picking based on factorswhich are automatically updateable in the quality control subsystem.

A robotic fruit picking system in which the end effector separates theedible and palatable part of a ripe fruit from its stem or stalk withoutcontacting the edible part.

A robotic fruit picking system in which the system automatically gradesa fruit by size and other measures of suitability that are programmed into, or learnt by, the QC subsystem.

A robotic fruit picking system in which the system automaticallytransfers a picked fruit to a suitable storage container held in thestorage subsystem without handling the edible and palatable part of thefruit or other sensitive parts of the fruit that could be bruised byhandling.

A robotic fruit picking system in which the control subsystem minimisesthe risk of the end effector or other part of the robot damaging a fruitor plant on which the fruit grows using machine learning based pickingstrategies.

A robotic fruit picking system in which the picking arm moves anattached camera to allow the computer vision subsystem to locate targetfruits and determine their pose and suitability for picking.

A robotic fruit picking system in which the picking arm is a lightweight robotic arm with at least some joints that exhibit a range ofmotion of +/−275 degrees that positions the end effector for picking andmoves picked fruit to the QC subsystem.

A robotic fruit picking system in which the control subsystem operatesthe total positioning system and the picking arm.

A robotic fruit picking system in which the control subsystem uses inputfrom the computer vision subsystem that analyses fruit images to decidewhere and when to move the robot.

A robotic fruit picking system in which the QC subsystem is responsiblefor grading picked fruit, determining its suitability for retail orother use, and discarding unusable fruit.

A robotic fruit picking system in which the robot picks rotten orotherwise unsuitable fruit (either by accident or design), and thendiscards that fruit into a suitable container within the robot or ontothe ground, and that container is accessible via a discard chute withits aperture positioned at the bottom of the QC rig so that the arm candrop the fruit immediately without the need to move to an alternativecontainer.

A robotic fruit picking system in which positive or negative airpressure is induced in a discard chute or an imaging chamber (e.g. usinga fan) to ensure that fungal spores coming from previously discardedfruit are kept away from healthy fruit in the imaging chamber.

A robotic fruit picking system in which the system comprises one or more6-axis light weight robotic picking arms with some or all joints thatexhibit a range of motion of +/−275 degrees.

A robotic fruit picking system in which the system comprises two or morepicking arms and the picking arms are positioned asymmetrically on therobot.

A robotic fruit picking system in which the robot has tracks that areremovable and the robot can run on rails if the tracks are removed.

A robotic fruit picking system in which the robot is equipped with fruitholding trays that are suspension mounted.

A robotic fruit picking system in which the robot is equipped with fruitholding trays that are mounted on movable arms that move from a firstextended position to a second, more compact position.

A robotic fruit picking system in which the robot is equipped with fruitholding trays arranged as two or more vertically oriented stacks.

A robotic fruit picking system in which the robot is powered from aremote power source.

A robotic fruit picking system in which the robot has one or more lights(e.g. strobe lights) that activate when a fruit tray or holder needs tobe replaced.

A robotic fruit picking system in which a fast moving robot removestrays or holders automatically from a slower moving robot that does thefruit picking.

A robotic fruit picking system in which the robot has one or more lights(e.g. strobe lights) that activate in response to a user input and shinean identifying signal above the robot.

A robotic fruit picking system in which the system includes an imagingor analysis chamber in which fruit is placed by the picking arm and isthen imaged or analysed for grading or quality control purposes.

A robotic fruit picking system in which the system includes an imagingor analysis chamber in which fruit is imaged or analysed for grading orquality control purposes and in which the imaging or analysis chamberincludes an aperture and a chimney or cylinder or lid or baffle on topof the imaging or analysis chamber's aperture that is designed to blockunwanted light from entering the chamber whilst still permitting fruitto be lowered or passed into it.

A robotic fruit picking system in which the system includes an imagingor analysis chamber in which fruit is imaged or analysed for grading orquality control purposes and in which the imaging or analysis chamberincludes one or more cameras and/or other sensors, such as camerassensitive to specific (and possibly non-visible) portions of the EMspectrum including IR, (ii) cameras and illuminators that use polarisedlight, and (iii) sensors specific to particular chemical compounds thatmight be emitted by the fruit.)

A robotic fruit picking system in which the system includes a cablemanagement system for cables that run through the articulating joints ofa robot, the cable management system including a cable enclosure and acentral cable guide that twists relative to the enclosure, allowing acoil or spiral of cable to expand and contract as the joints rotate.

The robot may include a picking arm made up of several individual rigidbodies, each attached to another rigid body at an articulating joint,and there is a cable enclosure associated with one or more of each ofthe articulating joints. The cable guide may be configured such that thecable is ducted away through the centre of the cable enclosure to thenext body. The cable management system may be configured to minimise thechange of local curvature of the cable as the articulating joints movethrough their full range of motion. The cables may be unscreened and theenclosure then provides screening. The cables may also serve to providesufficient heat to reduce cable degradation

A robotic fruit picking system in which the picking arm is adjustableand can be repositioned to maximize picking efficiency for a particularcrop variety or growing system, such as for the height of a specifictable top growing system.

A robotic fruit picking system in which the system is configured toperform several functions in addition to picking, including the abilityto spray weeds or pests with suitable herbicides and pesticides, or toreposition or prune trusses to facilitate vigorous fruit growth orsubsequent picking.

A robotic fruit picking system in which the robot estimates its positionand orientation with respect to a crop row by measuring its positionand/or orientation displacement relative to a tensioned cable.

A robotic fruit picking system in which the robot estimates its positionand orientation with respect to a crop row by measuring its displacementrelative to a tensioned cable that runs along the row (a ‘vectorcable’).

A robotic fruit picking system in which the robot includes one or morefollower arms that are mounted to follow the robot.

A robotic fruit picking system in which the follower arm is connected atone end to the robot chassis by means of a hinged joint and at the otherto a truck that runs along the cable.

A robotic fruit picking system in which the angle at the hinged joint ismeasured to determine the displacement relative to the cable.

A robotic fruit picking system in which the angle is measured from theresistance of a potentiometer.

A robotic fruit picking system in which two follower arms are used todetermine displacement and orientation relative to the vector cable.

A robotic fruit picking system in which a computer vision guidancesystem measures the displacement of the robot relative to the vectorcable.

A robotic fruit picking system in which the computer vision systemmeasures the projected position of the cable in 2D images obtained by acamera mounted with known position and orientation in the robotcoordinate system.

A robotic fruit picking system in which a bracket allows the vectorcable to be attached to the legs of tables on which crops grow.

A robotic fruit picking system in which a truck is equipped with amicroswitch positioned so as to break an electrical circuit with thetruck loses contact with the cable.

A robotic fruit picking system in which magnetic coupling is used toattach an outer portion of the follower arm to an inner portion of thefollower arm, such that in the event of a failure or other event theportions of the follower arm can separate without damage.

A robotic fruit picking system in which the separation of an outerportion and an inner portion of the follower arm triggers a controlsoftware to stop the robot.

Operational Optimization Features

In this section, we summarise features which contribute to theoperational effectiveness of the system.

A robotic fruit picking system in which the picking arm is controlled tooptimize trade-off between picking speed and picking accuracy.

A robotic fruit picking system in which the control sub-systemdetermines the suitability of a specific target fruit or bunch forpicking via a particular approach trajectory by determining thestatistical probability that an attempt to pick that target will besuccessful.

A robotic fruit picking system in which estimating the probability ofpicking success is determined from images of the scene obtained fromviewpoints near a particular target fruit.

A robotic fruit picking system in which determining the statisticalprobability is based on a multivariate statistical mode, such as MonteCarlo simulation.

A robotic fruit picking system in which the statistical model is trainedand updated from picking success data obtained by working robots.

A robotic fruit picking system in which the control sub-systemdetermines the probability of collision between a picking arm and anobject using an implicit 3D model of the scene formed by the range ofviewpoints from which a target fruit can be observed without occlusion.

A robotic fruit picking system in which the control sub-systemdetermines one or more viewpoints from which the target fruit appearsun-occluded, and hence identifies an obstacle free region of space.

A robotic fruit picking system in which the computer vision subsystemuses a statistical prior to obtain a maximum likelihood estimate of thevalues of the shape parameters of a target fruit and the system thencalculates an estimate of the volume of that target, and from thatestimates the weight of the target.

A robotic fruit picking system in which the computer vision subsystemdetermines the fruit's size and shape.

A robotic fruit picking system in which the computer vision subsystemdetermines the fruit's size and shape as a means of estimating thefruit's mass and thereby of ensuring that the require mass of fruit isplaced in each punnet according to the requirements of the intendedcustomer for average or minimum mass per punnet.

A robotic fruit picking system in which picked fruit is automaticallyallocated into specific punnets or containers based on size and qualitymeasures of the picked fruit to minimize the statistical expectation oftotal cost according to a metric of maximizing the expectedprofitability for a grower.

A robotic fruit picking system in which a probability distributiondescribing the size of picked fruits and other measures of quality isupdated dynamically as fruit is picked.

A robotic fruit picking system in which the picking arm places largerstrawberries in punnets that are more distant from the base of thepicking arm, in order to minimize the number of time-consuming arm movesto distant punnets.

A robotic fruit picking system in which the picking arm places selectedfruits in a separate storage container for subsequent scrutiny andre-packing by a human operator, if the quality control subsystemidentifies those selected fruits as requiring scrutiny by a humanoperator.

A robotic fruit picking system in which the control subsystem implementsa two-phase picking procedure for bunches of fruit, in which first theentire bunch is picked and second, unsuitable individual fruits areremoved from it.

A robotic fruit picking system in which the robot measures the pose ofthe picked fruit, so that the fruit can be positioned at the optimalpose for imaging or analysis or for release at the optimal height tofall into a punnet or container.

A robotic fruit picking system in which the robot determines theposition of other picked fruit already in a punnet or container andvaries the release position or height into the punnet or containeraccordingly for new fruit to be added to the punnet or container.

A robotic fruit picking system in which the robot automaticallypositions or orients picked fruit in a punnet or other container tomaximize visual appeal.

A robotic fruit picking system in which the robot automaticallygenerates a record of the quality or other properties of a fruit in aspecific punnet and adds a machine readable image to that punnet that islinked back to that record.

A robotic fruit picking system in which the robot chooses paths withinfree regions of the ground so as to distribute routes over the surfaceof the ground in a way the optimizes the trade off between journey timeand damage to the ground.

A robotic fruit picking system in which a chain of several robotsautomatically follow a single ‘lead’ robot that is driven under humancontrol.

A robotic fruit picking system in which information about the positionof several robots and the urgency of any fault condition, or impendingfault condition, affecting one or more robots is used to plan a humansupervisor's route amongst them.

A robotic fruit picking system in which the position of the robot withrespect to target fruit is controlled to optimize picking performance,such as minimizing expected picking time.

A robotic fruit picking system in which collision-free paths or obstaclefree trajectories are identified in advance of run-time by physicalsimulation of the motion of the robot between one or more pairs ofpoints in the configuration space, thereby defining a graph (or ‘routemap’) in which the nodes correspond to configurations (and associatedend effector poses) and edges correspond to valid routes betweenconfigurations.

A robotic fruit picking system in which a robot arm path planning isbuilt by mapping between regions of space (‘voxels’) and edges of aroute map graph corresponding to configuration space paths that wouldcause the robot to intersect that region during some or all of itsmotion,

A robotic fruit picking system in which the system logs undesirableconditions in the environment that might require subsequent humanintervention along with a map coordinate.

A robotic fruit picking system in which the system stores locations ofall detected fruit (whether ripe or unripe) in computer memory in orderto generate a yield map.

A robotic fruit picking system which the yield map enables a farmer toidentify problems such as disease or under- or over-watering.

A robotic fruit picking system in which the system stores a mapcoordinate system position of unripe fruits that have been detected butnot picked in computer memory.

A robotic fruit picking system in which the yield map takes into accountthe impact on time on the ripeness of previously unripe fruits.

A robotic fruit picking system in which the system measures the degreeto which the robot is leaning over and compensates for the degree oflean by adapting models of the scene's geometry and camera viewpointsaccordingly.

A robotic fruit picking system in which the system includes anaccelerometer.

A robotic fruit picking system in which the degree of lean is directlymeasured using the accelerometer or indirectly by measuring the positionof a part of the robot in a coordinate frame based on the crop row.

A robotic fruit picking system in which an appropriate 3D to 3Dtransformation required to correct for the lean is applied topre-defined camera poses and environment geometry.

A robotic fruit picking system in which the lateral position of therobot's tracks in the row is dynamically adjusted so that the pickingarms are closer to their design position despite the lean.

A robotic fruit picking system in which oscillations of a fruit causedby picking the fruit are reduced by damping achieved by a soft gripper.

A robotic fruit picking system in which oscillations of a fruit causedby picking the fruit are reduced by damping achieved by modulating theacceleration or velocity or movement of the robot arm's end-effector.

A robotic fruit picking system in which the system estimates the massand pendulum length of the fruit.

A robotic fruit picking system in which the system designs adeceleration or acceleration profile (dynamically or otherwise) requiredto minimize the amplitude or duration of oscillations.

End Effector Features

In this section, we summarise features which relate to the end effector;we refer to the end effector as the ‘end effector’.

A robotic fruit picking system in which the end effector uses at leastthe following phases:

(i) a selection phase during which the target fruit is physicallypartitioned or separated from the plant or tree and/or other fruitsgrowing on the plant/tree and/or growing infrastructure and

(ii) a severing phase during which the target fruit is permanentlysevered from the plant/tree.

A robotic fruit picking system in which the end effector includes ahook.

A robotic fruit picking system in which the fruit is moved away from itsoriginal growing position during the selection phase.

A robotic fruit picking system in which a decision phase is introducedafter the selection phase and before the severing phase.

A robotic fruit picking system in which the decision phase includesrotation of the fruit by its stem or otherwise.

A robotic fruit picking system in which the decision phase is used todetermine whether or not to sever the fruit, or the manner in which thefruit should be severed.

A robotic fruit picking system in which the selection phase is madereversible.

A robotic fruit picking system in which the reversibility isaccomplished by a change of shape of the hook.

A robotic fruit picking system in which the reversibility isaccomplished by movement or rotation of the hook.

A robotic fruit picking system in which the system is capable ofsimultaneously gripping and cutting the stem of a target fruit.

A robotic fruit picking system in which the system includes multiplepicking units located on a single multiplexed end effector.

A robotic fruit picking system in which multiple picking functions on apicking unit are driven off a single actuator or motor, selectivelyengaged by lightweight means, such as: electromagnets, an engaging pin,rotary tab, or similar.

A robotic fruit picking system in which a single motor or actuatordrives one function across all units on the head, selectively engaged bymeans such as: an electromagnet, an engaging pin, rotary tab, orsimilar.

A robotic fruit picking system in which the functions are driven bylightweight means from elsewhere in the system, such as using: a bowdencable, torsional drive cable/spring, pneumatic or hydraulic means.

A robotic fruit picking system in which the end effector pulls thetarget fruit away from the plant in order to determine the fruit'ssuitability for picking before the fruit is permanently severed from theplant.

A robotic fruit picking system in which the end effector comprises ahook with a dynamically programmable trajectory.

A robotic fruit picking system in which the end effector uses at leastthe following phases:

(i) a selection phase during which the target fruit is physicallypartitioned or separated from the tree and/or other fruits growing onthe tree and/or growing infrastructure;

(ii) a severing phase during which the target fruit is permanentlysevered from the tree; and

in which the selection and severing phases are performed by theactuation of a loop, wherein the loop diameter, position and orientationare programmatically controlled.

A robotic fruit picking system in which the end effector uses at leastthe following phases:

(i) a selection phase during which the target fruit is physicallypartitioned or separated from the tree and/or other fruits growing onthe tree and/or growing infrastructure;

(ii) a severing phase during which the target fruit is permanentlysevered from the tree; and

and in which the selection and severing phases are performed by a set ofjaws, wherein the jaws diameter and position are programmaticallycontrolled.

A robotic fruit picking system in which the jaw attitude such as opened,partially closed or closed is programmatically controlled.

Computer Vision Features

In this section, we summarise features which relate to the computervision system used for autonomous navigation and the computer visionsubsystem used for fruit imaging.

A robotic fruit picking system in which the computer vision based systemis used in order to determine heading and lateral positions of the robotwith respect to a row of crops using images obtained by a forwards orbackwards facing camera pointing approximately along the row.

A robotic fruit picking system in which the computer vision basedsubsystem detects a target fruit and in which the robot includes an endeffector wherein part of the end effector is used as an exposure controltarget.

A robotic fruit picking system in which the control system software useslighting conditions inferred or derived from the weather forecast as aninput to the control subsystem or computer vision subsystem to controlpicking strategies or operations.

A robotic fruit picking system in which the computer vision basedsubsystem detects a target fruit and a end effector is able tophysically separate a candidate fruit further from the plant and otherfruits in the bunch before picking.

A robotic fruit picking system in which mirrors are positioned andoriented to provide multiple virtual views of the fruit.

A robotic fruit picking system in which the computer vision based systemobtains multiple images of a target fruit under different lightingconditions and infers information about the shape of the target fruit.

A robotic fruit picking system in which the computer vision basedsubsystem uses an image segmentation technique to provide an indicationof a fruit health.

A robotic fruit picking system in which the computer vision basedsubsystem detects the positions or points of the fruit achenes ordrupelets and assigns a cost to those positions or the arrangement ofthose positions using an energy function that assigns a lowest energy toregularly arranged positions.

A robotic fruit picking system in which a semantic labelling approach isused to detect achenes, such as a decision forest classifier.

A robotic fruit picking system in which the sum of costs over pointsprovides an indication of fruit health.

A robotic fruit picking system in which the indication of the fruithealth is provided from analysing one or more of the following: colourof the achenes, colour of the flesh of the fruit or 3D shape of thefruit.

A robotic fruit picking system in which a neural network or othermachine learning system, trained from a database of existing images withassociated expert-derived ground truth labels, is used.

A robotic fruit picking system in which the computer vision basedsubsystem is used to classify a fruit, and in which the system allows agrower to adjust thresholds for classifying the fruit.

A robotic fruit picking system in which the computer vision basedsubsystem locates specific parts of a plant or specific plants thatrequire a targeted localized application of chemicals such as herbicidesor pesticides.

A robotic fruit picking system in which the computer vision basedsubsystem detects instances of specific kinds of pathogen such as:insects, dry rot, wet rot.

A robotic fruit picking system in which the computer vision subsystemdetects drupelets or achenes using specularities induced on the surfaceof a fruit by a single point light source.

A robotic fruit picking system in which an end effector is used forpicking and another end effector is used for spraying.

A robotic fruit picking system in which an end effector is a sprayingend effector that contains a small reservoir of liquid chemical.

A robotic fruit picking system in which the picking arm visits a stationon the chassis to gather a cartridge of chemicals.

A robotic fruit picking system in which the picking arm visits acartridge in the chassis to suck the required liquid chemicals from acartridge into its reservoir or to expel unused chemical from itsreservoir back into a cartridge.

A robotic fruit picking system in which several different types ofchemical are combined in dynamically programmable combination to achievemore optimal local treatment.

A robotic fruit picking system in which multiple cartridges containmultiple different chemical combinations.

AI/Machine Learning Features

In this section, we summarise features which relate to AI or machinelearning features.

A robotic fruit picking system in which a machine learning approach isused to train a detection algorithm to automatically detect a targetfruit.

A robotic fruit picking system in which the system identifies fruit inRGB color images obtained by a camera.

A robotic fruit picking system in which the system identifies fruit indepth images obtained by dense stereo or otherwise.

A robotic fruit picking system in which the training data is a datasetin which the position and orientation of target fruit is annotated byhand in images of plants that are representative of those likely to beobtained by the camera.

A robotic fruit picking system in which a detection algorithm is trainedto perform semantic segmentation on the images captured by the camera.

A robotic fruit picking system in which the semantic segmentation labelseach image pixels such as ripe fruit, unripe fruit or other object.

A robotic fruit picking system in which a clustering algorithmaggregates the results of the semantic segmentation.

A robotic fruit picking system in which the machine learning approach isa decision forest classifier.

A robotic fruit picking system in which the machine learning approach isa convolutional neural network.

A robotic fruit picking system in which a convolutional neural networkis trained to distinguish image patches containing a target fruit attheir center from image patches that do not.

A robotic fruit picking system in which a sliding window approach isused to determine the positions of all images likely to contain a targetfruit.

A robotic fruit picking system in which a semantic segmentation is usedto identify the likely image locations of a target fruit for subsequentmore accurate classification or pose determination by a CNN or otherform of inference engine.

A robotic fruit picking system in which a machine learning approach witha regression model is used to predict the angles describing theorientation of approximately rotationally symmetric fruit from images,including monocular, stereo and depth images.

A robotic fruit picking system in which a machine learning approach isused to train a detection algorithm to identify and delineate stalks inimages captured by a camera.

A robotic fruit picking system in which a machine learning approach isused to train a prediction algorithm to predict how much improvement toan initial pose estimate for a target fruit is likely to be revealed bya given additional viewpoint.

A robotic fruit picking system in which the system predicts whichadditional information, including which viewpoints out of a set ofavailable viewpoints, is likely to be the most valuable, including themost beneficial to overall productivity.

A robotic fruit picking system in which the additional information isthe location or point where the stalk attaches to the target fruit.

A robotic fruit picking system in which the additional information isthe knowledge that the fruit is visible without occlusion from aparticular viewpoint.

A robotic fruit picking system in which the additional information isthe knowledge that the space between the camera and the fruit is free ofobstacles from a particular viewpoint.

A robotic fruit picking system in which the system recovers a 3D shapeof a target fruit from one or more images of the target fruit obtainedfrom one or more viewpoints, and in which a generative model of thetarget fruit's image appearance is used.

A robotic fruit picking system in which a geometric and/or photometricmodel fitting approach is used to predict the surface appearance of atarget fruit as well as the shadows cast by the target fruit onto itselfunder different, controlled lightning conditions.

A robotic fruit picking system in which the cost function, namely themeasure of agreement between images, is made robust to occlusion or thefruit is physically separated from sources of occlusion.

A robotic fruit picking system in which a machine learning approach isused to train a labeling algorithm to automatically assign a label to animage captured by the system, wherein pre-labeled images provided byhuman experts are used to train the system.

A robotic fruit picking system in which labelling data provided by humanexperts is used to train a machine learning system to assign qualitylabels automatically to newly picked fruit, by training an imageclassifier with training data comprising (i) images of the picked fruitobtained by the QC subsystem and (ii) associated quality labels providedby the human expert.

A robotic fruit picking system in which the control policy subsystem istrained via reinforcement learning while the robot is operating.

A robotic fruit picking system in which the control subsystem is trainedvia reinforcement learning, and wherein training is done by simulatingthe movements of the robot using images of the real-world environmentcaptured amongst available viewpoints.

A robotic fruit picking system in which the control system is trained topredict picking success via reinforcement learning, and in whichtraining is done in a simulated picking environment.

A robotic fruit picking system in which the system is trained to predictpicking success via reinforcement learning, in which a predictor ofpicking success is to ensure that the predicted path of the end effectorsweeps through a 3D volume that encompasses the target stalk but notother stalks.

A robotic fruit picking system in which the control subsystem is trainedvia reinforcement learning that includes actions carried out by humanoperators.

A robotic fruit picking system in which a machine learning approach isused to train a model to predict yield forecast.

A robotic fruit picking system in which the system records a mapcoordinate system location along with an image of all detected fruit andthe recorded data is used to train a model to estimate a crop yieldforecast.

A robotic fruit picking system in which the system uses picking successdata obtained by working robots to learn and refine the parameters of adynamically updateable statistical model for estimating picking successprobability.

Picking Process Features

In this section, we summarise features which relate to the pickingprocess used by the system.

A robotic fruit picking system in which the system is operable to cutthe stem of a fruit, in which the fruit is picked by first severing andgripping its stem, and the body of the fruit is removed from its stem ina subsequent operation.

A robotic fruit picking system in which the system is operable to severthe fruit from its stem using a jet of compressed air, without requiringthe handling of the body of the fruit.

A robotic fruit picking system in which the robot includes a collar,shaped to facilitate forcing the body of the fruit off the stem.

A robotic fruit picking system in which the system is operable to severthe stem from its fruit by using the inertia of the body of the targetfruit to separate the body of the fruit from its stem.

A robotic fruit picking system in which the system is operable to cut orsever the stalk of a fruit using a reciprocating back-and-forth motionof the fruit in the direction approximately perpendicular to its stalkor an oscillatory rotary motion with an axis of rotation approximatelyparallel to the stalk.

A robotic fruit picking system in which a path planning algorithm isused to model obstacles as probabilistic models of scene space occupancyby different types of obstacle with different material properties.

A robotic fruit picking system in which the end effector is operable tocut the stem of a fruit, in which the system includes a deformable endeffector that is designed to deform under compressive force.

A robotic fruit picking system in which the system is operable to cutthe stem of a fruit without handling the body of the fruit.

A robotic fruit picking system in which the robot is operable at nightwith a computer vision system that operates at night, and picks fruitwhen they are cooler and hence firmer to minimize bruising.

A robotic fruit picking system in which the end effector is operable tocut a fruit stem cleanly and without tearing to increase fruitproductivity.

A robotic fruit picking system in which the quality control subsystempredicts the flavour or quality of a fruit and places the fruit in aspecific storage container according to the flavour or qualityprediction.

A robotic fruit picking system in which the prediction of the flavour orquality of a fruit depends on the analysis of a growth trajectory datameasured over time for the fruit.

Methods or Applications

In this section, we summarise features which relate to methods orapplications of the system.

A method of optimizing fruit yield prediction by imaging each fruitusing the robotic fruit picking system defined above, to determineripeness or suitability for picking.

A method of optimizing fruit yield mapping across a fruit farm ormultiple fruit farms, by imaging each fruit using the robotic fruitpicking system defined above, to determine ripeness or suitability forpicking.

A method of maximizing fruit shelf life by using the robotic fruitpicking system defined above.

A method of selectively storing or punnetising the fruit with theoptimal flavor or quality by using the robotic fruit picking systemdefined above.

A final aspect is the fruit when picked using the robotic fruit pickingsystem defined above. The fruit can be strawberries, includingstrawberries grown on a table. The fruit can be raspberries. The fruitcan be apples, or pears, or peaches, or grapes, plums, cherries, orolives or tomatoes.

Note

It is to be understood that the above-referenced arrangements are onlyillustrative of the application for the principles of the presentinvention. Numerous modifications and alternative arrangements can bedevised without departing from the spirit and scope of the presentinvention. While the present invention has been shown in the drawingsand fully described above with particularity and detail in connectionwith what is presently deemed to be the most practical and preferredexample(s) of the invention, it will be apparent to those of ordinaryskill in the art that numerous modifications can be made withoutdeparting from the principles and concepts of the invention as set forthherein.

1. An agricultural robotic system comprising: two or more autonomousrobots, in which each robot is a crop harvesting or crop carrying orcrop spraying robot; a positioning subsystem configured to estimate ordetermine the position of each robot; a communication subsystem toenable the robots to automatically share information about theirposition such that the positioning subsystem enables autonomouspositioning of the several autonomous robots using a computerimplemented guidance system; and in which the robots are furtherconfigured to form a convoy of robots, all following a similar route. 2.The agricultural robotic system of claim 1, in which the two or morerobots form a convoy in which each robot follows a substantially similarroute using the computer implemented guidance system.
 3. Theagricultural robotic system of claim 1, in which each robot follows itspredecessor robot at a configurable target distance or after aconfigurable target time.
 4. The agricultural robotic system of claim 1,in which the positioning subsystem measures the displacement of eachrobot relative to a global coordinate frame or to other robots or thetrajectories of the other robots.
 5. The agricultural robotic system ofclaim 1, in which one or more robots automatically follow a single leadrobot.
 6. The agricultural robotic system of claim 5, in which thesingle lead robot is driven under human control.
 7. The agriculturalrobotic system of claim 5, in which the single lead robot is drivenunder human control using a user interface.
 8. The agricultural roboticsystem of claim 1, in which the system is configured to choose thetrajectory of each robot so as to distribute the trajectories of allrobots over the surface of the ground so as to optimize a configurablemetric.
 9. The agricultural robotic system of claim 8, in which thesystem is further configured to dynamically distribute the trajectories.10. The agricultural robotic system of claim 8, in which theconfigurable metric takes into account one or more of the following:journey time, distance, degree of spread or damage to the ground. 11.The agricultural robotic system of claim 1, in which the system includesa control subsystem that is configured such that each robot can achievea desired trajectory or pose relative to a trajectory or pose of thelead robot.
 12. The agricultural robotic system of claim 1, in which thesystem includes a graphical user interface that displays informationabout the several robots.
 13. The agricultural robotic system of claim1, in which a user interface provides control, such as start and stop,for each robot.
 14. The agricultural robotic system of claim 1, in whichthe system includes an emergency stop device configured to stop all therobots.
 15. The agricultural robotic system of claim 14, in which theemergency stop device is an emergency stop button and/or an emergencystop bumper.
 16. The agricultural robotic system of claim 1, in whichthe two or more robots include sensors, such as bumpers, that will causean emergency stop in the event that the robot encounters an obstacle.17. The agricultural robotic system of claim 1, in which the guidancesystem will emergency stop all robots if one robot emergency stops. 18.The agricultural robotic system of claim 1, in which a user interfaceprovides or displays an emergency stop button.
 19. The agriculturalrobotic system of claim 1, in which the communication subsystem usesWIFI or other wireless techniques.
 20. The agricultural robotic systemof claim 1, in which information about the several robots positionincludes time-stamped pose estimates obtained from each robot.
 21. Theagricultural robotic system of claim 1, in which the positioningsubsystem is also configured to estimate or determine the orientation ofeach robot.
 22. The agricultural robotic system of claim 1, in which theposition and orientation of each robot are relative to a map coordinatesystem.
 23. The agricultural robotic system of claim 1, in which thesystem is configured to estimate absolute pose of the several robots ina world coordinate system.
 24. The agricultural robotic system of claim1, in which the system is configured to estimate pose of each robotrelative to one or more neighbouring robots.
 25. The agriculturalrobotic system of claim 1, in which one or more cameras are attached toeach robot with a known pose in a standard robot coordinate frame. 26.The agricultural robotic system of claim 1, in which the positioningsubsystem includes sensors such as ultrasound sensors, accelerometers,forwards or backwards facing cameras.
 27. The agricultural roboticsystem of claim 26, in which information from one or more sensors isfused with information from a GPS positioning subsystem.
 28. Theagricultural robotic system of claim 1, in which the information aboutthe position of several robots and the urgency of any fault condition,or impending fault condition, affecting one or more robots is used toplan a human supervisor's route amongst them.
 29. The agriculturalrobotic system of claim 1, in which the robots are configured to havedistinctive features or markers for automatic detection using a computervision system.
 30. The agricultural robotic system of claim 29, in whichthe distinctive features or markers include a bar code, a QR code,combination of light or light pattern.
 31. The agricultural roboticsystem of claim 1, in which each robot includes a visual identifierwhich can be turned on or off by selecting the specific robot on a userinterface.
 32. The agricultural robotic system of claim 31, in which thevisual identifier includes a light or light pattern.
 33. Theagricultural robotic system of claim 31, in which the visual identifieris projected upwards onto the roof of a polytunnel in which crops arebeing grown.
 34. The agricultural robotic system of claim 1, in whichcouplings, such as mechanical couplings, are used to couple each robotto its predecessor robot.
 35. The agricultural robotic system of claim34, in which each robot includes a device to measure the direction andmagnitude of force being transmitted by a robot's coupling to itspredecessor such as to derive a control signal for its motors.
 36. Theagricultural robotic system of claim 34, in which the two or more robotsshare responsibility for providing motive force.